search

NVlabs / FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
https://nvlabs.github.io/FoundationPose/
Other
1.01k stars 112 forks source link

Misalignment Issue with Custom Data from Isaac Sim Camera, model-free #49

Closed shingarey closed 1 month ago

shingarey commented 1 month ago

Firstly, I want to express my gratitude for releasing this library.

I'm currently facing an issue with misalignment by inference. I recorded custom data from an Isaac Sim camera, gathering both image and depth data from the ROS2 topics. I then used bundlesdf to create a mesh and ran python bundlesdf/run_nerf.py on the data.

However, when I started run_demo.py, I noticed the misalignment. I'm not sure why this is happening. Have you encountered this issue before or do you have any idea why this might be occurring?

Mesh from bundlesdf: Screenshot from 2024-04-11 12-04-05

Video of run_demo.py:

https://github.com/NVlabs/FoundationPose/assets/22908950/4795ddb1-0484-425e-ac66-2db7621c2655

shingarey commented 1 month ago

It turns to be out, that I encountered a problem when using the model generated by run_nerf.py located in model/model.obj. For some reason, it doesn't seem to be well-reconstructed: Screenshot from 2024-04-11 12-58-24

However, when I switched to the mesh generated by bundlesdf (see post above), the issue was resolved.

@wenbowen123, do you have any insights as to why the model in model/model.obj appears so strange?

wenbowen123 commented 1 month ago

Hi, when you run the run_nerf.py from this repo, how did you provide the camera poses? Is it from object to camera or opposite?

wenbowen123 commented 1 month ago

And have you tried with the model-free pipeline on LINEMOD/YCB-Video dataset to get reasonable results? The steps are provided in readme.

shingarey commented 1 month ago

Hi, when you run the run_nerf.py from this repo, how did you provide the camera poses? Is it from object to camera or opposite?

Hi, I used the poses provided by BundleSDF, which are defined as obj_in_cam. So, the inverse of these poses should be used in cam_in_obj, correct?

Could you suggest the best method to define cam_in_obj for each image without utilizing the BundleSDF library? Thank you.

wenbowen123 commented 1 month ago

yes they are the inverse.

you can use other SLAM methods to get the camera poses, if you dont want to run BundleSDF. However, BundleSDF has been tested with FoundationPose many times and turns out to work the best. Is there any reason you dont want to use BundleSDF?

shingarey commented 1 month ago

There's no particular reason for not wanting to use it. It just took me some time to install, and I was wondering if there might be a more lightweight alternative. However, once installed, it functions well.

ssapsu commented 3 weeks ago

@shingarey Hello shingarey,

I'm curious about how you've managed to use ROS2 with conda? If it's okay with you, may I ask about it?