search

NVlabs / FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
https://nvlabs.github.io/FoundationPose/
Other
1.15k stars 134 forks source link

Follow-up on Issue 83: Tracking drifting towards the end fo the samples #119

Open KomputerMaster64 opened 1 month ago

KomputerMaster64 commented 1 month ago

This is in continuation of the Issue #83

the bottom two images show a significant jump which is not very suitable for tracking. You can increase the camera frame rate if possible. Otherwise, you can also try with higher iter

https://github.com/NVlabs/FoundationPose/blob/fce37c86be33ee49cf0b46ee29ed26d613521d6d/run_demo.py#L21

Hi @wenbowen123

I am sorry for the delayed response. I ran the experiments with values for track_refine_iter variable ranging from 2 to 50, however it was observed that the tracking invariably drifted towards the end samples (last 15 to 20 percent of the video). I would be delighted to share the files.

Regards, Prabhav

__ Edit: I am sharing the failure case for the case with track_refine_iter variable set to 50.

Remarks: For the shown case, the tracking failure happends towards the end of the samples. The tracking is optimal till image 1714129813249.png i.e. for the first 410 samples (Total of 619 image samples).

Additional Remarks: This is an improvement it to the first case where track_refine_iter variable was set to 2, and the tracking was optimal only upto the 157 image samples.

I have uploaded the new FoundationPose_demo_05_eri-5_tri-50.zip file here (this is corresponding to the track_refine_iter variable set to 50) in the Google Drive Folder.

Question/Note: The total size of the debugging directories for the 49 experiments is 7 GB. It would be great if you could confirm if it is required to share all of the experiments' files.

__ The last frame with the correct tracking (top), and the frame after which the trakcing was not optimal (bottom). 1714129813249 1714129813295

wenbowen123 commented 3 weeks ago

I checked your logs. The object is moving too fast at 1714129813249. Your camera frequency is not fast enough.

KomputerMaster64 commented 3 weeks ago

Thank you for the response. I will check for better camera hardware. Do you recommend performing actions at a slower pace? I have been using the Depth Camera D435i - Intel RealSense. I will be testing FoundationPose with the Depth Camera D455 - Intel RealSense.

wenbowen123 commented 3 weeks ago

did you check how much fps is the camera images? The upper bound of realsense is 30hz, but sometimes it could be lower than that. In the video, you are ejecting the cube to perform free-form sliding, that moment the motion is really big.