Transform poses in world frame to pixel coordinates

[aaguiar96]

Hello. I'm working on a mixed reality project with apriltags and I need to transform (x,y,z) poses to (u,v) pixel coordinates.

I've already use the P matrix from camera_info topic, but the pixel coordinates do not make sense. Can anybody help me?

Thank's in advance.

[fabrizioschiano]

How (and why) do you transform the (x,y,z) to (u,v). I would suggest to not do the transformation but to look into the code and understanding where the (u,v) coordinates are appearing.

I am working on apriltag2 so it could be slightly different from this version of the apriltag but in principle, the algorithm from a high-level perspective should work in the same way.

You need to find the moment in which the apriltag detects the square of the tag and in that moment you should have the coordinates (in the image plane, and therefore [u,v] coordinates) of 4 points which are the 4 vertexes of the tag. After retrieving these coordinates you can either stream directly those or just compute the 'center of gravity' of the u,v coordinates like: [(u1+u2+u3+u4)/4, (v1+v2+v3+v4)/4] = (u,v) Then the (u,v) correspond to the pixel coordinates of the center of the tag.

[aaguiar96]

Thanks for the help!

That makes a lot of sense. I'll try to do it that way. My only doubt is that i'm subscribing to apriltags node which returns [id, (x,y,z), quaternion] of each tag. So, aparently i dont have any information about the corners. I'll try to find a way to get that info.

Thanks again!

[fabrizioschiano]

Yeah, this is what I meant with : 'look into the code'.

Good luck and keep us updated.

[aaguiar96]

Hello again.

I've found the corners of the tags in pixels and i found a way to access those coordinates.

Thanks for your help, it was precious.

[fabrizioschiano]

I am happy my suggestion was useful. How did you do at the end?

I am just curious because I did this with the apriltag2 and I would like to know how does this work with this version :)

[aaguiar96]

I've created a publisher of the type std_msgs::Float64MultiArray in apriltag_detector.cpp file in apriltag_ros package. I found a FOREACH loop with the id and corner positions for each tag (detections variable).. So, in each iteration of this loop, I insert the tag id and the respective corners position and publish that result into another node.

As simples as this. Thanks again, André.

[jimmy-adams]

I've created a publisher of the type std_msgs::Float64MultiArray in apriltag_detector.cpp file in apriltag_ros package. I found a FOREACH loop with the id and corner positions for each tag (detections variable).. So, in each iteration of this loop, I insert the tag id and the respective corners position and publish that result into another node.

As simples as this. Thanks again, André.

Hi. Can you give me some tips how to use apriltag ros? will that be possible to detect tag on gazebo?

[aaguiar96]

Hi! I never used them with gazebo, but I don't see why not.

You simply have to:

• add the package to your workspace
• create a launch file similar to the example.launch where you subscribe to your camera messages and camera calibration properly
• create a node that subscribes, for example, to /tag_detections topic that has all the info about the detected tags

Hope it helps.

[jimmy-adams]

Hi! I never used them with gazebo, but I don't see why not.

You simply have to:

* add the package to your workspace

* create a launch file similar to the example.launch where you subscribe to your camera messages and camera calibration properly

* create a node that subscribes, for example, to `/tag_detections` topic that has all the info about the detected tags

Hope it helps.

Hi, Thanks for your idea. And i make it to get the tag detected with virtual camera from gazebo. Another question is that i use the sentence rospy.Subscriber("/tag_detections", Marker, detect_tag) to get info of the tag topic. But i am not sure about the second field Marker, is that correct? or should there be other variable or type. Also from the info detections:

• id: 2 size: 0.163513 pose: header: seq: 80 stamp: secs: 1484 nsecs: 39000000 frame_id: ardrone_base_bottomcam pose: position: x: 0.374098572979 y: 0.0511245243748 z: 0.608714751895 orientation: x: 0.612514737549 y: 0.790444242353 z: -0.00469190300599 w: 0.00125780104866 if i just want the center position of the tag on screen. which should i refer to?

Best Regards

[aaguiar96]

I've done it in C++ so I'm not sure if there are another arguments possibilities, but the published info seems fine. Now, you have the tag position in [X Y Z] world coordinates. To get [u v] pixel position of the tag you have two options: (1) apply the homography matrix to the world coordinates that converts them to pixels coordinates or (2) modify the apriltags node so that it publishes pixel pose instead of world coodinates. The second option is quite easy, if you analyze apriltags_detector you see that detection.cxy is what you need.

[jimmy-adams]

I've done it in C++ so I'm not sure if there are another arguments possibilities, but the published info seems fine. Now, you have the tag position in [X Y Z] world coordinates. To get [u v] pixel position of the tag you have two options: (1) apply the homography matrix to the world coordinates that converts them to pixels coordinates or (2) modify the apriltags node so that it publishes pixel pose instead of world coodinates. The second option is quite easy, if you analyze apriltags_detector you see that detection.cxy is what you need.

Hi! Thanks for your hint, it saves me a lot of time. Having read the above discussions, i wonder how can i also get the four corners' pixel coordinates? Best Regards