AprilTags Support on DepthAI/megaAI

[Luxonis-Brandon]

Start with the why:

In unstructured environments, DepthAI allows capabilities (spatial AI) which previously required multiple components (e.g. a depth camera, a host, and a neural processor) which then precluded application of this spatial AI in locations where embedded application is required (i.e. low size, weight, power, boot-time, and/or cost).

In structured environments (e.g. a warehouse), AprilTags (here) are the go-to, as they provide ‘cheat code’ level information on relative location (x,y,z) and pose (yaw, pitch, roll) of tags. So these can be put on objects (boxes) for locating them, walls or floors for navigation, or on complicated structures to know their pose/location/etc. real time.

So if DepthAI supports AprilTags, it allows use of this cheat code (fully offloaded from the host) in conjunction with the unique value-add that other solutions can’t do - unstructured 3D location of objects of interest (e.g. a human).

So then DepthAI could enable for example collaborative robotics, where a human (an unstructured object) is followed in a warehouse, and DepthAI is providing all the AprilTags results so that this robot knows exactly where it is in physical space while it follows the person. (And all sorts of other collaborative tasks you could imagine).

This sort of application is exactly why the evgrandprix (and other autonomous) 1/5th scale autonomous-racing competitions put AprilTags on their vehicles. It allows a cheat-code for each vehicle to know the relative location of the other vehicles in xyz and yaw/pitch/roll.

And most importantly, these tags are so ubiquitous used, there’s probably a whole slew of use-cases that we can’t even think of that folks will use DepthAI for if AprilTags are cleanly supported, particularly if at a high framerate. For example, Peter Mullen reached out this weekend for his application, here: https://april.eecs.umich.edu/software/apriltag, wanting April Tags support. His application is interesting as he is mounting them on walls such that the camera can self-localize, and he has a system (above) which actually uses the location in pixel space of the corners of the April Tags to further improve the localization results (doing SLAM-like techniques).

So we should include his suggested additional output (the location in pixel-space of the detected corners of the April Tags) detailed below in what:

The how:

Characterize how many resources AprilTags takes.

If AprilTags looks to ‘heavy’ for this, then define which stream(s) it needs to be mutually exclusive with. From their comment of “Real-time performance can be achieved even on cell-phone grade processors.” it seems like it will likely be runnable in parallel to the existing DepthAI functions.

• https://april.eecs.umich.edu/software/apriltag

The what:

Implement April Tag detection on DepthAI as a stream that users can optionally enable, ideally while the other current functions are being used (neural inference, depth, etc.).

Enable a user-selectable optional output of meta-data (in addition to the 6DoF result that AprilTags gives) which is the pixel-space coordinates of the 4 corners of the april tags (maybe as an optional output) correlated with the tag ID - allowing folks to do their own work off that data, either ignoring our 6DOF result from April Tags, or combining their own algorithm on top of it.

[Luxonis-Brandon]

As a quick update, initial support for this is in https://github.com/luxonis/depthai/pull/139.

[r1b4z01d]

Do you have an ETA for when this will be ready? Is there an example I can try now?

[Luxonis-Brandon]

Sorry about the delay. Let me ask internally. We had put this on hold in lieu of other features which had to date been more popularly requested.

[tomvdb]

Any updates? I kinda got this camera specifically for the april tag support it advertised :/

[Luxonis-Brandon]

Hi @tomvdb (and @r1b4z01d ),

So we are again working on this. @csaba-luxonis is working on it. That said, we understand that we did indeed market this feature and then not get it out, so we are more than happy to provide a full refund on your purchase.

If you'd like to do so, please feel free to reach out to us at support@luxonis.com.

Thanks, and sorry about the delay.

-Brandon

[tomvdb]

Hi Brandon,

Nope, not getting my oak-D back ;) All good, I'm starting to think that april tags/aruco tags might be better suited on the host side anyways. I've been doing some experiments and want to try and see if the depth features can enhance my aruco data on the host side.

Thanks!

[Luxonis-Brandon]

Thanks, @tomvdb . :-). Yes, AprilTags actually will run fairly well on a lot of hosts. We will be implementing it on DepthAI though as for pure-embedded usecases DepthAI is the only device that can run AprilTags.

@csaba-luxonis is on it. The last time we implemented it (https://github.com/luxonis/depthai/pull/139), the approach did not parallelize well, and we ended up just running it on the little CPU in the Myriad X instead. So we're starting from first principles this time to see if we can make it parallelize.

[GOBish]

HI @Luxonis-Brandon , any eta for when AprilTags will be ready for the Oak-D? I could really use it for my project. Thanks!

[Luxonis-Brandon]

So I know @csaba-luxonis was able to at least get it working on a single frame running on OAK-D. And then was working on sizing automatically from the cameras using Image Manip node. Will check in to see. I don't immediately have an ETA though.

[GOBish]

ok thanks!

[Luxonis-Brandon]

Here's the initial cut at it, only works on full-frame as of now @GOBish @tomvdb @r1b4z01d . There's a crash occurring with Image Manip node on RAW8 that is preventing it from working on lower resolutions/higher frame-rates.

https://github.com/luxonis/depthai-python/pull/298

[GOBish]

Thanks @Luxonis-Brandon ! Will give it a shot.

[Luxonis-Brandon]

Thanks, @GOBish ! And thanks goes to @csaba-luxonis , I'm just the messenger. :-)

[accio-robot]

Hello @Luxonis-Brandon! Is the support for full 6-DOF pose estimation available for OAK-D camera yet? I have found an AprilTag detection (not pose estimation) feature in the latest depthAI repository. Is there any way to get an update on the ETA for 'structured navigation using AprilTags' feature if it not already available?

[Luxonis-Brandon]

Hi @accio-robot ,

We don't have 6-DoF yet. I'm not sure on the effort to add that. I will ask internally.

[Erol444]

As of version 2.15, AprilTags support has been mainline to the depthai.

[madgrizzle]

I very much would like to see pose estimation of the april tag (I assume that's what the 6-dof is). I was planning on using an april tag to assist with the docking of my robot, but without the pose estimation it won't orient correctly.

[saching13]

Currently, you can use the returned points to run opencv's solvePnP on the host to get the 6 DOF. cc: @szabi-luxonis on the efforts to add this on device.

[SzabolcsGergely]

Currently, you can use the returned points to run opencv's solvePnP on the host to get the 6 DOF. cc: @szabi-luxonis on the efforts to add this on device.

Feel free to initially implement on the LEON cores, since it's not computationally heavy for that amount of parameters. We can look into porting to shaves once it's done, for extra performance, if needed.

[saching13]

Hhmm. I was thinking if we could just port the ePnP which you had tried already.

[SzabolcsGergely]

Hhmm. I was thinking if we could just port the ePnP which you had tried already.

ePnP is definitely easier to port, with Eigen implementation, otherwise the original uses opencv 1 or 2. Either way is fine for this problem.

[luisfico]

Hello Luxonis team and congratulations for your great job in OAK. I'm wondering if it's possible to know what is the detection algorithm used for the embedded AprilTag node currently? Is the detection method based on a specific branch of the official repository https://github.com/AprilRobotics/apriltag or maybe it uses a deep-learning approach?

[themarpe]

Thanks for the kind words @luisfico!

Yes, the AprilTag detection is based on the official repository, master branch.

[luisfico]

Thanks @themarpe for your assistance.

[MAVProxyUser]

Has there been an update on this? I tried out the example here: https://github.com/luxonis/depthai-python/blob/main/examples/AprilTag/apriltag.py

Is seems to perform poorly vs just using OpenCV with apriltag library. I notice for example it doesn't seem to do well with my test images at the same distances, or orientations. And more importantly it seems to be unable to give me a detection, unless there is some movement. After detecting it seems to not maintain a lock on the tag, unless it moves slightly.

Am I doing something wrong?

[MAVProxyUser]

This sample python code, that just accesses the camera image, and uses apriltag library would be my performance baseline. We don't seem to be close. Is there a way to get there?

import depthai as dai
import cv2
import apriltag

# Initialize AprilTag detector
detector = apriltag.Detector()

# Initialize Luxonis OAK-1-MAX pipeline
pipeline = dai.Pipeline()
cam = pipeline.createColorCamera()
cam.setPreviewSize(640, 480)
cam.setInterleaved(False)
cam.setFps(30)

# Define output stream
xout = pipeline.createXLinkOut()
xout.setStreamName("preview")
cam.preview.link(xout.input)

# Create device and start the pipeline
device = dai.Device(pipeline)
preview_queue = device.getOutputQueue(name="preview", maxSize=1, blocking=False)

while True:
    # Get the current frame from the camera
    frame = preview_queue.get().getCvFrame()

    # Convert frame to grayscale
    gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Detect AprilTags in the grayscale frame
    detections = detector.detect(gray_frame)

    # Print detected AprilTags information
    for detection in detections:
        print(f"Detected AprilTag: ID {detection.tag_id}, Center ({detection.center[0]}, {detection.center[1]})")

    # Display the frame with AprilTag detections (optional)
    for detection in detections:
        cv2.circle(frame, tuple(detection.center.astype(int)), 4, (0, 255, 0), 2)
        cv2.putText(frame, str(detection.tag_id), tuple(detection.center.astype(int)), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
    cv2.imshow('Preview', frame)

    # Exit the loop if 'q' key is pressed
    if cv2.waitKey(1) == ord('q'):
        break

# Clean up
cv2.destroyAllWindows()
device.close()
device = None

[MAVProxyUser]

I'm testing using these tags printed on 8x11 paper. Open CV + Apriltag lib does amazing... Luxonis cam + Onboard MyriadX, not so much, sadly.