search

arclab-hku / Event_based_VO-VIO-SLAM

Our Works in Event-based VO/VIO/SLAM
MIT License
226 stars 20 forks source link

Replicate the error metrics #10

Closed zeyulong closed 3 months ago

zeyulong commented 4 months ago

Thank you for your outstanding work.

I would like to replicate the error metrics in your paper. You mentioned in issue exact error metrics: in the IROS paper and PLEVIO, we use RMSE as error metrics.

However, when I downloaded the EIO in DAVIS346 results, and the RMSE error metric obtained by using the following command does not match the error metrics you provided:

evo_ape bag dvs_vicon_result.bag /dvs_vicon/gt_pose /pose_graph/imu_evio_loop -va -p

My method of testing error metrics must be wrong. You mentioned: the estimated and ground-truth trajectories were aligned with a 6-DOF transformation (in SE3), using 5 seconds [0-5s] of the resulting trajectory. The result is obtained through computing the mean position error (Euclidean distance in meters) as percentages of the total traveled distance of the ground truth. Unit: %/m (e.g. 0.24 means the average error would be 0.24m for 100m motion).

How did you get the error metrics? How is using 5 seconds [0-5s] of the resulting trajectory achieved?

Looking forward to your reply.

KwanWaiPang commented 4 months ago

Thanks for your interesting for our works. The trajectory alignment is the foundation for evaluating the accuracy of odometry and SLAM. I would like to suggest to get some basic knowledge of SLAM and the evaluation of SLAM. While I personally think the following might be useful. Hopefully, it can help you.

  1. Zhang, Zichao, and Davide Scaramuzza. "A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry." 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018.
  2. https://github.com/MichaelGrupp/evo