Inconsistent transformations between frames

[ivbelkin]

Hello!

In the window of Isaac Sim I see, that the camera installed below LiDAR, but when I look at the observations dict returned by benchbot api I see, that the pose of camera_left have approx z = 1.3m, and lidar z = 0.86. Is it ok?

Also, I'm using ROS for development and create simple visualization of the data in RViz, captured from benchbot (in container benchbot_roscore). In the picture below you can see frames (map, base_link, camera_right, lidar, camera_left). Height of camera_right looks ok, but camera_left is higher, than expected. Also, green boxes - are boxes taken from gt file - addons/benchbot_addons/benchbot-addons/ground_truths_bear_develop/ground_truths/miniroom_1.json. It can be seen, that all of them below real objects in camera data (look at alarm clock, flowerpot).

Looks like, there some problem in transformations, probably caused by migration to omniverse.

[david2611]

Hi @ivbelkin,thanks for your use of benchbot and for finding all of these issues that have come up during the transition to Omniverse.

First off, I can confirm that the digital camera is supposed to be that high. We use camera_left as the source of camera observations within our scene understanding challenge and set it up high in order to ensure the robot can see above tables etc. and find all relevant objects. More info on the robot setup (though admittedly within a paper for our Unreal version of BEAR) can be found in our BEAR data paper.

Regarding the vertical shift in the GT map that definitely appears to be a bug that we will look into ASAP. From your observations can you confirm that there is only a vertical shift of the GT boxes? In your image it appears that the clock might also have some translation error as well in the bottom half of your picture.

[btalb]

From our debugging, this seems to be what would be causing #46 as well.

There is a vertical shift error in the transforms output by the Carter simulation in Isaac. We will look at resolving and releasing a fix today.

[david2611]

Hi @ivbelkin, we think we have fixed this issue by updating our carter model in the latest robot_sim_omni commit.

Assumptions being made about the position of base_link w.r.t. the odom frame had been made incorrect by the transfer to omniverse. You can access these updates by either pulling that specific repo in your add-ons folder, or by performing fresh benchbot_install

Would you be able to confirm for us that this fix solves your problem?

[ivbelkin]

Thank you! It is much better now! There is no vertical shift, however, looks like some translation error is also presented, as @david2611 noticed

[david2611]

Hi @ivbelkin, thanks for sending through the updated info and apologies for the delay. Unfortunately at present we have not been able to replicate your problems on our end. Going into our Omniverse environments we have been able to confirm that the object bounding boxes align correctly within Omniverse (minus the sink in miniroom:1 which was a good catch thank you :+1: ). See images below for visual confirmation of bounding boxes within Omniverse.

If you had been using any of the code from the Semantic SLAM example in creating your visualizations, it may be fixed now after we caught a bug in the use of camera intrinsics in #53. If this does not help you with your issue, would it be possible for you to send through the code you are using for your provided visualizations to help us replicate on our end?

[ivbelkin]

Hi, @david2611! After fixing camera intrinsics I still get some displacement( Moreover, small vertical shift returns. I create my visualizations in rviz, probably, the data published from simulator to ROS have some problems

Here what I do step by step:

1. Start simulator in the 1st console: ./benchbot_run --robot carter_omni --env miniroom:1 --task semantic_slam:passive:ground_truth
2. Start hello_passive example in the 2nd console ./benchbot_submit --example hello_passive
3. Enter into container benchbot_robot in the 3d console docker exec -it benchbot_robot bash and start recording ROS messages to bag file: rosbag record -O miniroom_1 /camera/camera_info /camera/color/image_raw /camera/depth/image_raw /cmd_vel /odom /point_cloud /scan /tf /tf_static
4. Press ENTER in the 2nd console
5. After passive_example have finished I stoped rosbag record and extract it from container to my host by docker cp
6. Then, to make this bag visualizable by rviz I did some fixes to the header.frame_id and topic names using this script
7. To publish ground truth as ROS message I use this script

This is initial bag file, and this is after using script at 6 step

[david2611]

Hi @ivbelkin thanks for all the info. This has been very helpful in our debugging process.

Can I check if you have tested this situation on more than just miniroom:1? Thus far our testing seems to indicate that it is present in miniroom environments but not house ones. Have you also found the same to be true on your end?

[ivbelkin]

Hi @david2611! I checked house env and there is no such problem, i.e. what robot sees is well aligned with ground truth

[david2611]

Bug was found to be a mathematical issue in the provided starting pose for the Miniroom environment. Quaternion provided was mathematically invalid causing a large error in rotation of a few degrees which made the camera projections offset by a few degrees. This has been updated for all miniroom environments as well as some house environments and challenge environments. A fresh benchbot_install should install the updated add-ons and sort out this issue. It has been a long time since this post was last interacted with for which we apologize. If you find this has indeed fixed the issue for you or there is no interaction within a week we shall mark this as closed. Below are some examples from miniroom:1 with the updated start pose and we attach the rosbag recording used to generate them.

[david2611]

Closing issue after time elapsed from seeming solution launch