after ransac, frame has too few matches #1

[kamzero]

Hi, I’m trying to run BundleSDF on BEHAVE 30fps dataset. I use run_behave.py script and use config_behave.yml config file. However, while the object are moving fast, I often encounter the problem that loftr matches are too few after ransac. e.g. sequence Date03_Sub03_boxlarge.2 Is there anything I haven't noticed? Could you please share you hyperparameters of these sequences? Thanks for your time and consideration.

[kamzero]

[wenbowen123]

It is normal that in challenging datasets like BEHAVE, the matches could often be very limited. However, BundleSDF is robust to such scenarios. What if you remove the pdb and continue? Since this is a refactor of the original code, I haven't got a chance to test.

[kamzero]

Thank you for your reply. I am trying to continue running after pdb pause. The inaccurate pose estimation caused by failed matches on sequences like Date03_Sub03_chairwood_lift may seriously affect the reconstruction quality. However, on simpler sequences like Date03_Sub3_boxlarge, it seems to have little effect on reconstruction quality (since I have not evaluated the pose estimation yet, I am not sure if the error in pose estimation would significantly increase at such times).

I have noticed an interesting phenomenon that on the Date03_Sub3_boxlarge sequence, compared to moments of motion blur caused by high-speed translation, match failures seem to occur more frequently when the rotation is larger. I would like to ask which viewpoint you agree with more: that the matching accuracy of loftr is limited and cannot meet the geometric constraints, or that the parameters related to rotation in RANSAC need to be adjusted?

[wenbowen123]

match failures seem to occur more frequently when the rotation is larger

for this I'd suggest to tune the RANSAC parameters first. If there are textures on the objects, loftr usually work well even under large rotations. Also make sure the segmentation quality is not too bad.

[kamzero]

I further analyzed the situation of "few matches after RANSAC" during rotation and found that it was not due to a lack of matches in LOFTR, but rather because few matching points remained after RANSAC. Upon observing several frames of BoxLarge, I found that this often occurred when a face of BoxLarge appeared or disappeared in the camera's field of view due to rotation. This reminded me of the observability problem of rotation angles in visual odometry research. Have you analyzed your algorithm from this perspective?

[wenbowen123]

can you elaborate more on the meaning of your viz and the problem you mentioned?