search

NVlabs / FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
https://nvlabs.github.io/FoundationPose/
Other
1.47k stars 196 forks source link

Set up a camera in Isaac Sim to work with FoundationPose #129

Closed grth25 closed 5 months ago

grth25 commented 5 months ago

Hi, I've seen in your Paper that you have worked with Isaac Sim so I hope that you might help me with my problem. I am currently trying to get images from a camera set up in Isaac Sim and use them to estimate the pose of objects. I am using the realsense camera from "franka_alt_fingers.usd" but the results I get from Pose estimation are not convenient. I have already tested the Pose Estimation with the object I want to estimate in reality with a Realsense D435 camera and it worked really well. But for my project I need to test it in Isaac Sim.

I am not quite sure how the image data should be converted to work well with FoundationPose. I have seen in other issues that the depth images need to be in uint16 and in milimeter scale.

This is my program to set up the camera parameters and get the rgb and depth images:

assets_root_path = get_assets_root_path() asset_path = assets_root_path + "/Isaac/Robots/Franka/franka_alt_fingers.usd" add_reference_to_stage(asset_path, "/Franka") camera = Camera("/Franka/panda_hand/geometry/realsense/realsense_camera", resolution=(640,480)) my_world.reset() camera.initialize() camera.set_horizontal_aperture(0.1) camera.add_distance_to_camera_to_frame() camera.add_distance_to_image_plane_to_frame()

intrinsics = camera.get_intrinsics_matrix() rgb = camera.get_rgb() rgb_img = PIL.Image.fromarray(rgb) rgb_img.save(f"./images/rgb/{framename}.png") depth = camera.get_depth().astype(np.uint16) depth = depth*1000 depth_image = PIL.Image.fromarray(depth) depth_image.save(f"./images/depth/{framename}.png")

I believe that the intrinsics matrix is not quite correctly and that the scale of the depth image might not be right, but I am not sure how I should edit it.

Intrinsics matrix: 12.351999282836914 0.0 320.0 0.0 6.947999477386475 240.0 0.0 0.0 1.0

Screenshot 2024-05-16 163539

Do you have an idea how I could edit the camera data to get correct results in Isaac Sim?

Thank you very much for your help.

grth25 commented 5 months ago

I solved the problem by editing the camera parameters (https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/isaac_sim_sensors_camera.html?highlight=rational#calibrated-camera-sensors) and changing the order depth = camera.get_depth()*1000 depth = depth.astype(np.uint16)