3D position of objects using new Kinect

[changliu6]

The 3D positioning of object using Kinect seems to rely on openni, which does not support Kinect One. Is there a workaround for this issue?

By the way, I'm running Kinect One on a Windows PC and use k2_client and k2_server to publish Kinect data (rgb, depth, ir, body) to ROS on a Ubuntu laptop. Can find_object_2d extract 3D position of an object using only these data topics?

[matlabbe]

It can support any RGB-D camera in ROS. The example uses openni_launch package, but it could use iai_kinect2 package as well for Kinect v2 (Xbox One). In your case, k2_client could also be used. Just remap the correct topic names to RGB and depth inputs of find_object_2d.

cheers, Mathieu

[changliu6]

Thanks Mathieu!

I remapped the topics to what k2_client provides: /rgb/image_rect_color -> /head/kinect2/rgb/image_color /depth_registered/image_raw -> /head/kinect2/rgb/image_depth (for find_object_3d.launch) /depth_registered/camera_info -> /head/kinect2/rgb/camera_info (for find_object_3d.launch)

However, neither find_object_2d.launch nor find_object_3d.launch works correctly.

find_object_2d.launch This is the result by using find_object_2d.launch: monitor at the top shows rgb image and monitor at the bottom shows the output in find_object_2d gui. The image in gui is not shown correctly and gets updated very slowly.

When running

rosrun find_object_2d print_objects_detected

Nothing is printed out.

By the way, terminal sometimes shows error "No objects loaded from /home/vdl/objects" (red sentence in the screenshot). Does this mean the program expects pre-saved object image for recognition?

find_object_3d.launch When running find_object_3d.launch, gui sometimes gives similar image as the above one, but takes longer time to generate. But for most of time, it gets stuck like this:

Do you have any ideas what may go wrong? Kinect v2 sends more data than the previous version, could this be the problem?

[matlabbe]

Hi,

Make sure the images are correctly published (use rqt_image_view for convenience).

For convenience, I created an example launch file for the kinect2 (using iai_kinect2): find_object_3d_kinect2.launch

Like the example on ROS wiki, you can do:

$ roslaunch kinect2_bridge kinect2_bridge.launch publish_tf:=true
$ roslaunch find_object_2d find_object_3d_kinect2.launch

cheers

[adarshmodh]

Hey I'm using your library for kinect_one with iai_kinect2 but I'm always getting the same warning:

[ WARN] [1465205005.292113246]: Lookup would require extrapolation at time 1465205004.967803836, but only time 1465204992.926743625 is in the buffer, when looking up transform from frame [object_5] to frame [map] [ WARN] [1465205005.520634991]: Object 5 detected, center 2D at (346.778093,437.321170), but invalid depth, cannot set frame "object_5"! (maybe object is too near of the camera or bad depth image)

I am not able to get the [x,y,z] in the map frame or the optical frame........ so can you suggest something to get those values for the object that is getting detected...........

[matlabbe]

Hi,

The warning tells that there is are missing depth values in the depth image in the object detected. To be more robust, we could modify the code to look for depth values around the detected object's points of interest used to compute the orientation of the object. See the code here, the three points should have valid depth to compute the transform.

cheers

[Abduoit]

Hi matlabbe

I am trying to run these codes $ roslaunch kinect2_bridge kinect2_bridge.launch publish_tf:=true $ roslaunch find_object_2d find_object_3d_kinect2.launch

but I don't get 3d image like yours, I only get 2d image, btw I don't know how to use find_object_3d_kinect2.launch I mean how to run your example find-object/launch/find_object_3d_kinect2.launch

could u plz help me????

[matlabbe]

Can you link where you see "3d image like yours"? find_object UI shows only 2d images. If you mean RVIZ, then use DepthCloud display for convenience: http://wiki.ros.org/turtlebot/Tutorials/indigo/3D%20Visualisation

kinect2_bridge.launch launches the kinect driver and publish rgb/depth images. find_object_3d_kinect2.launch subscribes to rgb/depth images from kinect, detect objects in 2d with rgb image, then uses depth image to get where are the objects in 3D, accordingly to camera frame. As stated in the launch file, you should not use the static transforms for /map and/odom frames on a final system, they are just there for the example (in a real system, /map and /odom would be published by localization and odometry modules respectively). tf_example node is another example of subscribing to find_object output topic to know when an object is detected, then use TF to get its pose.

cheers, Mathieu

[Abduoit]

Thanks to your reply, I am using kinect v2 with ros ubuntu 16.04. When I try to run (roslaunch kinect2_bridge kinect2_bridge.launch publish_tf:=true) and (roslaunch find_object_2d find_object_3d_kinect2.launch), I only get, like this pic I don't get like the following pic, I am planning to work on 3d construction [may be by using rviz] i.e detecting objects with depth information. Could u plz tell me how can I get like the following pic.

[matlabbe]

Outputs are similar, did you add an object to detect? You can open the Objects and Parameters panels under menu View. To add a new object, do "Edit->Add object from scene...".

When an object is detected, there would be some messages in the terminal as in the top of the screenshot.

cheers, Mathieu

[Abduoit]

Thanks for your fast response

I got the following pic, but I still can't understand the msgs in terminal, and I realize that detection process is very weak, i.e the particular object could be easily no longer being detected once I move it a bit. I appreciate your time.

[matlabbe]

2D local features have their limitations, as they work better with highly textured objects like book covers. You won't get good detection or tracking with this kind of objects.

If you want to track objects like that, you may want to try OpenTLD: https://github.com/zk00006/OpenTLD or an approach similar.

cheers, Mathieu

[Abduoit]

Thank you so much for your reply, u r saving my time

I tried OpenTLD, I used webcam,

but I still don't know how to use it with kinect v2 and how it works, whats the concept behind it???

thx again

[TrinhNC]

Hi, I'm using Kinect v2 and running this example to detect an object:

$ roslaunch kinect2_bridge kinect2_bridge.launch publish_tf:=true
$ roslaunch find_object_2d find_object_3d_kinect2.launch

However when I run rviz and add TF, I don't see any object there. The picture is as below. Do you know what's the problem?

[matlabbe]

Did you add an object in find-object's UI? If so, look at the terminal for warnings if the object is detected in the find-object's UI and not shown in RVIZ.

[LynnRobot]

Hi matlabbe,

I'm using the 3D sensors of the robot Pepper. Because of the poor resolution, I changed the resolution of the 2D sensors to 640x480, the depth resolution has to be 320x240 or lower. After changing the settings I'm getting the same warnings:

• Object detected, center 2D at (...), but invalid depth, cannot set frame (maybe object is too near of the camera or bad depth image)
• Object passed to lookupTransform argument source_frame does not exist.

Is there a possibility to solve this problem? Thanks a lot!

[matlabbe]

Hi,

Look at the code computing the 3D position: https://github.com/introlab/find-object/blob/2cb9120f7424d2fba13932dc8755b57acb2814a7/src/ros/FindObjectROS.cpp#L53

It takes 3 points: center, a point on x axis and another one on y axis of the object. Get the depth at those points and compute the 3D pose with those points. If depth is invalid on any of these points, the warning appears and TF is not published.

You could modify the code to be more robust to sparse depth (taking more points on x and y axes for example). As for the depth resolution, the code assumes that rgb and depth are the same size. You may adjust the code to divide by two the positions of the points before getting their corresponding depth.

cheers, Mathieu

[hygxy]

@matlabbe can you please elaborate on "It takes 3 points: center, a point on the x-axis and another one on the y-axis of the object. Get the depth at those points and compute the 3D pose with those points", or give a link regarding the theory?

[matlabbe]

Here is the whole code, which would be straightforward to follow: https://github.com/introlab/find-object/blob/2cb9120f7424d2fba13932dc8755b57acb2814a7/src/ros/FindObjectROS.cpp#L82-L131

In the screenshot below, I represented the three points used in red, then the derived axes in light blue. To get the third axis, we just do a cross multiplication between the two axes.

[hygxy]

@matlabbe Two things that I still can't understand: 1, Why is the y-axis then inverted? 2. Why is the rotation matrix constructed in that way(the DCM matrix is used, or?), hope you can also help me with that, thanks

[matlabbe]

1. In the screenshot the y-axis is wrongly drawn, it should go down (in camera coordinates) like in the code.
2. As we have 3 normalised and perpendicular vectors for each axis, we can just create the rotation matrix using these vectors.

[shrijan00]

@matlabbe So i can track my object using Kinect with the help of find_object_2d. After tracking how can we get the pose of the desired object?

[matlabbe]

find_object_2d doesn't do tracking, it just detects the object in a frame, there is no temporal motion prediction between frames. You can subscribe to objectsStamped topic, and when an object is detected, lookup its pose with TF. Look at this example: https://github.com/introlab/find-object/blob/master/src/ros/tf_example_node.cpp

If you didn't try the kinect yet, try this example: http://wiki.ros.org/find_object_2d#A3D_position_of_the_objects

[shrijan00]

@matlabbe So i want to track an object (for ex a ping pong ball) and get pose of it bwhich is placed on a AGV(automated guided vehicle) which is moving very slowly like 1 m/s. find_object_2d wont be efficient right. I observed open_tld was more useful. So i was using freenect_launch for my kinect instead of openi and tried the same thing but didn´t get the pose.

[shrijan00]

I got the pose by doing rosrun ¨tf tf_echo /camera_depth_frame /target frame¨ But i´m still not sure whether to go for find_object_2d, blob tracking or open_tld.

[matlabbe]

For a ping pong ball, you will have a hard time detecting it with find_object. For open_tld, if you get a pose in the image frame (in pixels), you could find the corresponding depth of the ball using the registered depth image, then compute the pose in 3D yourself (knowing the depth and camera calibration). Note that you may look at first if the depth image has valid depth values on the ball, otherwise you may look for a stereo camera and compute disparity of the tracked object yourself.

cheers, Mathieu

[shrijan00]

@Matlabbe Thanks. I was trying to do with openCv and ROS as it is easy to get pose for Aruco so if that is not as accurate then i would surely go for this.

[shrijan00]

@matlabbe Hey, i was able to track the object properly so to get the pose of the object from fixed stereo camera, should i use tf to get the data?

[matlabbe]

tf is a tool to share transforms between ros nodes, tf won't compute the pose for you. Note that to use a stereo camera like the kinect example, you should generate a depth image from the stereo, then use it like a RGB-D camera.

[akhilmasetti1]

can i use this package to integrate with arm can i expect a good result?

[manaswinimotheram]

@akhilmasetti1 have u tried that one? even I am using the same thing

[manaswinimotheram]

@matlabbe

sir can u please help me!!! i got strucked in this sir,im getting some data can i rely on them i need to integrate with arm

by trhe way im using kinect v2 camera there some issue in the launch file of find_object_3d.launch file

[matlabbe]

The code would build on arm, though never did myself. The project is depending on OpenCV and Qt, so if both can be built on arm, find-object should build too. For kinect v2 under arm, you will have to ask on iai_kinect2 repo.

[zahrakhabir]

Hi I want to use the find_object_2d package in the simulation environment(Gazebo). My problem is that it does not work on changing topic in find_object_3d.launch .I know I have to do a series of changes in the node, but I don't know which node choose. I use turtlebot2 in Gazebo. Thank you for your attention

[matlabbe]

You would just have to change those topics based on what you have with tutblebot2: https://github.com/introlab/find-object/blob/f7c38f91d0b076f365592974ff75972aa668b7f1/launch/find_object_3d.launch#L13-L15

[zahrakhabir]

hi my project is about mobile robot localization with ekf. The Kinect sensor is used as an extroceptive sensor to detect landmarks. For this purpose, I was first able to identify landmarks using the find_object_3d package, but I could not subscribe to the data which were the coordinates of the detected landmarks in the observation step. I would really appreciate it if anyone helped me on how I could use this information in the observation step.

[matlabbe]

Do you need a 3D pose of the object? To do so, there is an example here using TF to get that pose when an object is detected. https://github.com/introlab/find-object/blob/master/src/ros/tf_example_node.cpp

[zahrakhabir]

yes, but i can not use data (tf_example.node) for the observation step in EKF . When I subscribed the ObjectsStamped topic, I don't see equal data. I need 3D pose of the object for extended kalman filter on real time. Please tell me what to do. Thank you.

[matlabbe]

The pose is known here: https://github.com/introlab/find-object/blob/f7c38f91d0b076f365592974ff75972aa668b7f1/src/ros/tf_example_node.cpp#L84 you would have to convert it to format you want (e.g., geometry_msgs::PoseWithCovarianceStamped) and republish it. You would have to arbitrary set the covariance, as this info is not available.