search

wenbowen123 / BundleTrack

[IROS 2021] BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models
Other
603 stars 66 forks source link

Bundletrack loses tracking #55

Closed ChrisSun99 closed 1 year ago

ChrisSun99 commented 1 year ago

Hi @wenbowen123, thank you for your nice work. I've been trying to run Bundletrack on my own RGBD data (linked here), but I wasn't able to get a good result. I attached our previous results below. Do you have any intuition of problems with the data? Or is it possible for you to achieve any better results with this data? Thank you. Figure_1 Figure_2 Figure_3 Figure_4

wenbowen123 commented 1 year ago

Hi, the depth images are not in the right format, can you save them as uint16 and mm scale? From a skim of the data, it looks like the robot arm moves relatively fast. So make sure the max_rot and max_trans >= the max possible velocity https://github.com/wenbowen123/BundleTrack/blob/3df16853c745e7f216ea45df386a57fff0fc9c39/config_ycbineoat.yml#L59

Also increase the LOG allows for more detailed visualization (see some old issues for discussions on this) to check what's wrong https://github.com/wenbowen123/BundleTrack/blob/3df16853c745e7f216ea45df386a57fff0fc9c39/config_ycbineoat.yml#L6

ChrisSun99 commented 1 year ago

Thank you for your reply! What are the meaning of the four relevant parameters? Are their units "degree" and "meters" correspondingly? I'm trying to play around with their numbers but got no luck.

wenbowen123 commented 1 year ago

The translation are meters, the rotations are degree.

The most important thing is the depth image you uploaded here was wrong. If you fixed them, but still had problems. This issue might be helpful for debugging. There are visualizations and output to check where went wrong.

ebianchi commented 1 year ago

Hey Bowen -- working with @ChrisSun99 here. Thanks for your debugging help so far. We've got more information for you to help figure out where we are going wrong. Here are 6 things we will get to you -- @ChrisSun99 will send the data along shortly, but here's the high-level overview of what and why we are providing:

  1. Our updated data, now that it's in the right format of uint16 values.
  2. Log files generated by changing LOG: 3 as you suggested.
  3. After updating our data, we will show the new ground truth versus BundleTrack pose estimate plots, like that which we first posted.
  4. A couple screenshots of the masked RGB image at a couple of the high movement timestamps. Would you be able to intuit if these RGB images look too blurry for BundleTrack to be able to handle?
  5. Confirming reference frames with you: We provide the first pose of the object in the camera frame, the frame in which BundleTrack reports the rest of the object poses. Our plots above of the tracking are in world frame, so we transform BundleTrack's output back into the world frame to include them on our plots.
  6. More of a conceptual question about how BundleTrack works: Is it possible for BundleTrack to predict poses that are far away from the provided mask of the object? For instance, in the original plots we provided at timestamp ~900, the x- and y- differences are over half a meter (I know those original plots were made with data in the wrong format, but the updated plots (3) that @ChrisSun99 will provide still show similarly big variation). If BundleTrack is biased towards predicting poses that are close to the provided mask, then that would make me believe any large differences in our plots (3) might indicate a problem with (5) not successfully comparing to ground truth in the same reference frame.
ChrisSun99 commented 1 year ago
  1. Here is the updated data.
  2. Here is the 100th frame which looks good to me. Another example is frame 506. But I do noticed there are some intermediate frames (like this) where RANSAC fails.
  3. Here are some plots. Figure_1

Figure_2 c2762-4aea-443d-9819-d1c201e26faa.png) Figure_3 Figure_4

  1. 0255_match_0251_BA 0265_match_0255_BA

wenbowen123 commented 1 year ago

Hi @ChrisSun99 @ebianchi , there are multiple challenges in you data: (severe motion blur, errorneous segmentation, fast rotations, and far distance from camera).

I'd recommend slower your robot arm motion if possible and also mount the camera closer to the workspace. There also seems to be some constant shift in the segmentation.

@ebianchi point 5 sounds reasonable to me. But some left/right multiplication can be common pitfalls. https://github.com/wenbowen123/BundleTrack/issues/38 discusses about this. Starting from sim would be a nice idea.

For point 6, I'm not sure if I fully understand about the shift. The mask should ideally cover the correct object region in each image. Then bundletrack will provide you the right poses. It is fine if the object move to different locations in the image along the video (as long as segmentations of each frame are in the ballpark). The current segmentation looks poor. In some images, it's like more than half size of the object away from the true mask.

Sorry I'm fully occupied in the coming month. I can check more closely your data details afterwards (thanks for your patience)

wenbowen123 commented 1 year ago

Hi @ebianchi @ChrisSun99 , it seems your plot is not correct. I visualized the point cloud by transforming their poses and they are reasonably aligned. The first image is the point cloud of first frame. The second is the the merged point cloud of first frame and the 200th frame. But in your plot, the 200 time stamp is already deviating a lot. image image

ChrisSun99 commented 1 year ago

@wenbowen123 Thank you very much! It turned out our transformation was incorrect. We followed #38 and the result looks reasonable now.