gazebosim / gz-sensors

Provides numerous sensor models designed to generate realistic data from simulation environments.
https://gazebosim.org
Apache License 2.0
63 stars 58 forks source link

Pointcloud constructed from rgbd_sensor depth+rgb image has different orientation to points output #29

Open heuristicus opened 4 years ago

heuristicus commented 4 years ago

I'm using the subt stack for the reproduction because I already have it set up. Given that this is an issue with the sensor itself I think this would also reproduce in a basic setup with just a camera.

To reproduce:

You can use this rviz config file to reproduce the rviz setup I have. You will have to change the depth map topic of the DepthCloud to an invalid topic, like /explorer_x1/front_rgbd/image_raw and then change it back to /explorer_x1/front_rgbd/depth/image_raw in order for the depth cloud to display correctly. I do not think this weirdness is connected to the issue as I have displayed the pointcloud in another visualisation tool and it displays the same behaviour.

The issue

The rgdb_sensor provides the topics sensor/image_raw, sensor/depth/image_raw and sensor/points. When displaying the sensor/points pointcloud in rviz the cloud appears in the expected orientation.

The cloud that is displayed by the DepthCloud generated by combining the depth and rgb points is in a different orientation to that which is displayed by the pointcloud display, despite the fact that they should be in the same orientation.

From the images above it appears that the cloud generated from the depth/rgb image expects that the positive z-axis points out into the world from the camera lens, whereas the points cloud assumes that is the case for the positive x axis.

This problem is not present in gazebo. I believe this is because the rgb and depth images are published in a separate frame from the base frame of the camera, the sensor optical_frame, where the positive z axis is in the expected orientation: Screenshot from 2020-06-04 11-30-12

Here you can see the incorrect orientation of the clouds relative to each other:

Screenshot from 2020-06-04 11-15-38 Screenshot from 2020-06-04 11-12-59

realsense_frame

iche033 commented 4 years ago

for subt, we recently added an optical frame so that DepthCloud can be visualized with the desired orientation, see osrf/subt#413.

As for visualization not showing up the first time, this comment in the pull request describes a workaround to make the visualizations appear.

heuristicus commented 4 years ago

Thanks for that information - it fixes the issue for me. An upstream fix on the sensor itself would still be preferable though.