search

qcr / benchbot

BenchBot is a tool for seamlessly testing & evaluating semantic scene understanding tools in both realistic 3D simulation & on real robots
BSD 3-Clause "New" or "Revised" License
110 stars 12 forks source link

Out of sync rgb and depth images #12

Closed david2611 closed 2 years ago

david2611 commented 3 years ago

There are instances (at least in the visualization) where the RGB and depth images appear to be out of sync.

It appears to be an issue of synchronization rather than rgb and depth coming from cameras with different intrinsics. This is proved by the below gif of the rgb and depth after a move_distance 0.0 command, showing the images are perfectly aligned. depth_rgb_comp_d0 0

The severity of this effect appears to be amplified by how quickly the robot was moving before it stops as after some motions, there is still perfect overlap. This is shown below with the results of a 90 degree rotation where the robot smoothly slowed down to a stop: depth_rgb_comparison_a_90_smooth

The depth image appears to be ahead of the rgb image (at least in my couple of tests) as shown below.

Rotating anticlockwise (+ve 20) depth_rgb_comparison_a_20

Rotating clockwise (-ve 20) depth_rgb_comparison_a_-20

This issue is not limited to rotation commands as after a large distance command, due to the robot motion in the simulator, sharp motion may be required to correct orientation.

Move distance 0.7 depth_rgb_comp_d0 7_rough

Note that because of the correction being a very sharp rotational movement, the difference is more severe than in previous examples wherein the robot still moved quite smoothly.

Again I cannot confirm yet whether this is an issue with the visualizer or the entire system of sending messages but it should be looked into so we can guarantee users are provided with perfectly aligned RGBD information.

david2611 commented 3 years ago

Looking at the move distance example again, from memory the motion of snapping into place was a clockwise motion which suggests the depth might not always lead the rgb

btalb commented 3 years ago

We found the issue seemed to be caused from deeper down in the stack (Isaac Simulator level).

Need to debug deeper down the stack to understand the cause of the issue.

btalb commented 2 years ago

Closing for now due to moving to the new simulator (Omniverse-powered Isaac Sim).

Will re-open if we can reproduce the issue in Omniverse.