Poor performance on fast-moving data & inconsistent orientation on constant pose estimations

[shzcuber]

Hello,

I've noticed that the tracking performance seems to deteriorate with fast motion as shown in this video, even after raising my track_refine_iter to 100:

https://github.com/NVlabs/FoundationPose/assets/60201722/e31a292f-85ba-40f2-8a5b-df9922d07598

I have a workaround by running the pose estimation more often to recalibrate the bounding box. However, this means the orientation will become inconsistent:

https://github.com/NVlabs/FoundationPose/assets/60201722/dbd52249-9fbc-46d4-bd44-66cddd256147

Any advice on how to fix either of these issues? Thanks!

[wenbowen123]

Hi, it does not look like tracking issue. Even in the first frame, it does not look correct. There is likely something wrong with your depth/object textured mesh/mask. I'd suggest to search among the issues, since quite some are related.

[shzcuber]

Got it. Are you able to look at my data and let me know what the issue is? We were thinking there may be things wrong with our depth output since the output visually isn't totally black (unlike the images in the output example data), but we used the script in #44 for that, so I'm not sure how likely that is. Additionally for the mesh, we only have a .obj file and no .mtl files or texture files — is it possible that this is the issue?

https://drive.google.com/drive/folders/1bTsTsxpalcK_JtnvTo4MYKf_jFNOmif2?usp=sharing

[wenbowen123]

Like you said, I think both the depth and model have issues. Addressing these should make it much better.

[shzcuber]

Can you provide more details on what's wrong with the depth and the model? The depth seems to be in uint16 mm format. Do we need to add the mtl texture files for the obj?

[wenbowen123]

For object, how did you get the obj file? You need textures on it. You can use tools such as BundleSDF

For depth, refer to https://github.com/NVlabs/FoundationPose/issues/112#issuecomment-2100622913 to check the ply file.