Question: possible to use depth-ai with other open-cv detection like aruco markers?

[whogben]

I've just gotten my hands on an OAK-D and I'm hoping to use it to provide 3d locations of aruco markers to aid in precise robot navigation, eg trying to plug itself into an electrical outlet.

I've been following this tutorial for aruco detection with regular cameras - and am curious if it's possible to use this with depthai to get the 3d positions directly. I checked the docs on OpenVINO and the OpenVINO Model Zoo, but didn't find anything about running other types of opencv models.

Please let me know if there's an easy way to run this on the OAK-D, or if I should be using another approach (eg, running aruco detection on each mono frame separately and triangulating on the host computer).

[jonititan]

It definitely is as evidenced by their use of aruco boards during factory calibration. Code link

[Luxonis-Brandon]

Hi @whogben ,

So we have initially ported AprilTags (see here), but it's in a VERY old branch at this point. We have been meaning to make it a node in our current Gen2 API but have been behind, and specifically microPython with threading support (needed to be able to use on multiple nodes) has been a LOT more timeconsuming than we thought.

That said, in the implementation of AprilTags in DepthAI/OAK-D, we realized that the algorithm is set up extremely well for CPU operations and therefore does not benefit really at all from the massively-parallel capabilities of the Myriad X.

So actually running it on the host is often advised (unless you have a really slow host).

So what I would recommend is actually to run the AruCo on the host on the rectified_right frame, which is 1:1 aligned with depth. As then with this, you can actually just use the crop-node to pull back the depth corresponding from the ROI of the AruCo marker (to reduce the amount of data having to go to the host).

Does that make sense?

And I'm thinking you could use Image Manip node to pull out all the depth of the AruCo ROI, or you could use the spatial location calculator node instead so you literally just get the XYZ of the AruCo ROI center.

See this example here: https://docs.luxonis.com/projects/api/en/latest/samples/27_spatial_location_calculator/

So you just pass the pixel coordinates of the AruCo detected on the rectified_right image (or could be just right image, not 100% sure) and it will spit back to you the XYZ location of that ROI.

Thoughts?

And our engineers can probably do a better job than me advising, and feel free to join our Discord (here) to get real-ish-time help from the engineering team and our development community at large.

Thanks, Brandon

[marcelino-pensa]

Hi, @Luxonis-Brandon! Are there any examples on how to run apriltags on the Oak cameras (I would be looking into the OAK-1, in specific)? As you mentioned, I could run it in my host machine, but it is resource-constrained. I want to have multiple cameras pointing at different directions while looking for apriltags, and I would greatly benefit from having the software running onboard the device itself, as it would be really demanding to run on the host. Is the link you sent enough to run it on a OAK-1, or is it too old (as you mentioned) to easily integrate?

[Luxonis-Brandon]

Sorry about the delay.

Theoretically what I linked should run on OAK-1. It's just old now and so we haven't tried it very recently. But back then at least it did run.

We've just been behind - mainly bug squashing - so we have not gotten to getting AprilTags into Gen2 yet. And other features have been higher need by more folks, so that's why this hasn't gotten to the top of the queue yet.

Sorry about the delay and unfortunate news here.

[eric-schleicher]

video update, looks like this is inching along.

(click)

[EnzoBergamini]

Hello,

I am also working on the same project of getting 3d position of an aruco marker. I have a AOK-D lite camera and am wondering why did say to you use the right camera instead of the RGB centered camera, the color camera has a way better résolution so it can possibly have a better detection range.

Is there a reason to use the right camera for the detection ?