Tracking does not start even though AprilTags are visible

[mgagvani]

Hello,

I am trying to use the Spectacular AI SDK's AprilTag functionality in order to get pose estimates by viewing multiple AprilTags with known poses. Here is an example picture from the camera:

The tracking doesn't start until I move the camera around. Although, it should be able to get its pose given that there are 3 AprilTags in sight. How can I get the pose of the camera without needing to move the camera, given there is an AprilTag in view and config.aprilTagPath has been configured?

Also, does the SDK treat AprilTags as landmarks with known pose, or does it combine the corners/edges of the AprilTag with the other features being tracked?

[kaatrasa]

Hey,

1) The tracking doesn't start until I move the camera around...

Thanks for spotting this issue with the AprilTag mode; there should be a fix included in the next release (most likely later this week).

2) Also, does the SDK treat AprilTags as landmarks with known pose, or does it combine the corners/edges of the AprilTag with the other features being tracked?

AprilTags are used as landmarks for (re-)localization, but internally we do use the AprilTag corner points.

[mgagvani]

Would it be possible for the SDK to output the positions of the AprilTags it finds as part of the hooks API?

[kaatrasa]

Release 1.27 is now public & the initialization while stationary should hopefully be better now. (Also spotted another related AprilTag issue, but didn't have time to include the fix in this release, so if this version doesn't work yet, then hopefully the next release will).

Would it be possible for the SDK to output the positions of the AprilTags it finds as part of the hooks API?

We would like to output detected AprilTags through our Mapping API at some point, but currently it is not very high priority so cannot promise when this feature will be added.

[mgagvani]

I see. It would be extremely helpful at the very least to know if an AprilTag is currently being detected and its current ID, but having access to the poses of the AprilTags relative to the camera would be amazing.

[kaatrasa]

I totally undestand, and all of those features should eventually be accessible through the mapping API. However, unless a commercial project requires those features in near future, then I cannot promise when they will be added.

As a temporary solution, could you consider detecting the AprilTags yourself on the OAK-D? I know it's not ideal to detect the AprilTags twice, but this is probably the fastest solution for your use case. DepthAI seem to have their own AprilTag example (did not test): https://github.com/luxonis/depthai-python/blob/main/examples/AprilTag/apriltag.py

[mgagvani]

Thank you. I'm still getting some instances where the Tracking Status remains INIT even when there are AprilTags in the frame. This is with a more complex set of AprilTags, the JSON for which I am using is here: https://github.com/mgagvani/apriltag-localization-frc/blob/main/apriltag_configs/crescendo/crescendo_apriltags.json

Visualizing the tags in 3d, it looks like this:

I believe that the way I've set up the JSON file, the VIO output should start wherever the camera is relative to the AprilTag. However, what ends up happening is that if/when the tracking does start, it starts at the origin rather than the correct position. Here is an example image of the AprilTags I'm trying to detect:

[kaatrasa]

Very cool setup!

I'm still getting some instances where the Tracking Status remains INIT even when there are AprilTags in the frame.

We are aware that this wasn't entirely fixed with version SDK 1.27 and will try to improve it. I will let you know when fix is available.

I believe that the way I've set up the JSON file, the VIO output should start wherever the camera is relative to the AprilTag. However, what ends up happening is that if/when the tracking does start, it starts at the origin rather than the correct position.

I couldn't reproduce this issue; My tags.json looks like this

[
    {
        "id": 0,
        "size": 0.139,
        "family": "tagStandard41h12",
        "tagToWorld": [
            [    1,    0,    0,    10.0],
            [    0,    0,    1,    0],
            [    0,    -1,    0,    10.0],
            [    0,    0,    0,    1]
    ]
    }
],

i.e. tag with origin at [10, 0, 10] and mapping_visu.py example looks like this

[kaatrasa]

Hey,

1) We finally fixed the initialization issue with release 1.33.0. 2) In addition, April Tag observations are also available through mapping API. See https://github.com/SpectacularAI/sdk-examples/blob/main/python/oak/april_tag.py

To update the SDK to latest version, run pip install spectacularai[full] --force