MichaelGrupp
commented
1 month ago When you compare identity rotations to a ground truth with rotational RPE, you're measuring essentially the rotation angle of your ground truth between two poses within your chosen delta (default 1 frame).
For a very short delta or small motions in the ground truth, you might not see a big difference between your identities and a real trajectory. This does not mean that your dummy trajectory is "competitive", you're just measuring one specific thing. Look at RPE with higher delta (e.g. --delta 1 --delta_unit m) or absolute metrics and you will probably very quickly see that the data is of course completely wrong.
You need to check your data in detail yourself if you want to investigate this further.
dd2252
Hi, thanks for this great repo.
I'm running a sanity check baseline of random translation and identity rotation for the trajectories with camera pose on SINTEL. Over the 14 train-set sequences with non-trivial trajectory, the identity gets a mean RPE Rotation of 0.96 degrees, better than many methods like DROID-SLAM (1.91 degrees) or Robust Dynamic NeRF (1.31 degrees), and competitive with the state of the art (Particle SfM, 0.535 degrees). Given there is large rotation within the sequences of SINTEL, I find it hard to believe the identity is competitive with these methods; also 1 degree of rotation error doesn't seem correct for sequences that can rotate by 45 or more degrees.
I'm running the code from here under "Evaluation Results". I also tried this on other data and got a similar result: the identity is competitive with SOTA methods.
Is this expected? I.e. maybe RPE Rotation evaluates each frame pair sequentially, and other methods are not as consistent on a frame-to-frame basis.
Thank you!!