search

NVlabs / FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
https://nvlabs.github.io/FoundationPose/
Other
1.33k stars 173 forks source link

How does FoundationPose pick the first frame for the pose estimate? #186

Open sharanyashastry opened 2 months ago

sharanyashastry commented 2 months ago

Hi! This is some great work @wenbowen123! It's turning out to be very useful for my research.

I was wondering how the initial frame is estimated/placed? I was able to get it to run for a novel object using the model based method. I just want my initial frame to be rotated by 45 degrees in order to have it align in a more visually intuitive way with the object I'm tracking.

For example, what if I wanted this initial triad to be rotated in such a way that the blue axis aligns with the text and the red points at the cap but the green axis is not visible? image

wenbowen123 commented 1 month ago

you can change it in the mesh. For the OBJ file, you can open it in Meshlab and rotate it to the way you want, then over-write the original one.

sharanyashastry commented 1 month ago

Could you point me to where in the code the initialization is done?

wenbowen123 commented 1 month ago

there is no code needed for this. As I said, you just open the OBJ file in Meshlab and rotate it to the way you want, then over-write the original one.

sharanyashastry commented 1 month ago

Here's what my obj file looks like which is axis aligned and oriented the way I want. image

Here's what the tracking looks like: image

I would expect that this should have foundation pose pick an axis aligned initial pose. I'm not sure why this isn't the case - do you have any suggestions?

wenbowen123 commented 1 month ago

https://github.com/NVlabs/FoundationPose/blob/main/run_demo.py#L69 what if you change this line to 

center_pose = pose
sharanyashastry commented 1 month ago

I made the change and this is what happens (images from two runs with the same initial pose): image image image

From different initial pose: image

wenbowen123 commented 1 month ago

It seems like they are axis-aligned now? It's hard to tell from your mesh editing tool how the object is oriented.

sharanyashastry commented 1 month ago

It isn't consistent. It sometimes picks the initial pose to be axis aligned but not always. If you look at the first two images, blue and green axes are wrong.

Is it expected to differ with every experiment? If yes, is there a way to ensure it doesn't?

wenbowen123 commented 1 month ago

Are you able to share your data? I can take a closer look at it.