search

castacks / DytanVO

[ICRA'23] DytanVO: Visual Odometry in Dynamic Environments
BSD 3-Clause "New" or "Revised" License
173 stars 20 forks source link

Question about the ATE (m) of Droid-slam on KITTI #13

Closed LinMenwill closed 10 months ago

LinMenwill commented 1 year ago

Thank you for your outstanding work! While attempting to reproduce the results of Droid-slam on the KITTI through the provided pre-trained model, I have been unable to achieve results of comparable quality as yours. Could you share some evaluation details of droid-slam? Did you train it all over again on KITTI or just with the pre-trained model? Thanks a bunch!

SecureSheII commented 1 year ago

No, we used the same pre-trained model but we disabled the backend because we tried to only compare ourselves with the frontend of DROID (note that even with only the frontend, they still do local BA). So we commented out line 81 to line 85 below

https://github.com/princeton-vl/DROID-SLAM/blob/8016d2b9b72b101a3e9ac804ebf20b4c654dc291/droid_slam/droid.py#L79-L85

Also make sure you have the correct custom calibration file because KITTI Odometry dataset does not share the same intrinsics across all sequences.

We used the same TartanAir evaluator inside their repo to calculate the ATE.

LinMenwill commented 1 year ago

@Geniussh Thanks for your quick reply. I've been using the TartanAir evaluator from both your repository and the Droid-Slam repository to conduct tests. For sequence 10, the obtained results are as follows:

'ate_score': 25.296660981807168,
 'rpe_score': (0.014398422163911627, 32.556640976664816), 
'kitti_score': (0.002590124581328853, 0.08948751360026441)

whether the ATE metric mentioned in Table 2 corresponds to the 'kitti_score' rather than the 'ate_score'?

SecureSheII commented 1 year ago

All tables in the paper are ATE scores. I cannot reproduce what your score on my end. Make sure you use the correct intrinsics per KITTI sequence and follow the format required by Droid. Plus, what data did you use to evaluate Droid? Is it our Dynamic KITTI or the original KITTI odometry dataset?