Up to date ATE and RTE metrics

[mateosss]

Hello, are there any up to date reports of the system performance on euroc and tumvi datasets? I wasn't able to find such numbers, only the ones from the original 2020 paper which I imagine might have changed.

Thank you.

[goldbattle]

I don't have any, but the script you can use to generate them for your hardware are in the repo: ov_msckf/scripts/

[mateosss]

After a while, I got to compare OpenVINS against Basalt on the EuRoC datasets using the xrtslam-metrics tooling. I used the default euroc settings, and ran it without ROS. Here are the results for anyone that might find it useful (you can cite the Monado SLAM datasets if needed).

Note that I removed the EMH04 dataset from the average row since OpenVINS diverged on it.

Average completion percentage [%]

(Percentage of the dataset for which the system gives a trajectory estimate to evaluate)

	Basalt	OpenVINS
EMH01	100%	75.60%
EMH02	100%	74.30%
EMH03	100%	85.59%
EMH04	100%	94.39%
EMH05	100%	81.34%
EV101	100%	96.15%
EV102	100%	94.21%
EV103	100%	93.30%
EV201	100%	95.66%
EV202	100%	96.51%
EV203	100%	92.89%
[AVG]	100%	88.55%

Absolute trajectory error (ATE) [m]

	Basalt	OpenVINS
EMH01	0.076 ± 0.021	0.070 ± 0.037
EMH02	0.048 ± 0.024	0.111 ± 0.044
EMH03	0.060 ± 0.022	0.125 ± 0.054
EMH04	0.126 ± 0.047	12427.358 ± 6462.951
EMH05	0.145 ± 0.043	0.200 ± 0.089
EV101	0.045 ± 0.018	0.061 ± 0.021
EV102	0.054 ± 0.017	0.080 ± 0.038
EV103	0.056 ± 0.014	0.059 ± 0.026
EV201	0.036 ± 0.013	0.048 ± 0.026
EV202	0.040 ± 0.014	0.042 ± 0.018
EV203	0.156 ± 0.059	0.117 ± 0.035 *
[AVG]	0.072 ± 0.024	0.091 ± 0.039

Relative trajectory error (RTE) [m]

	Basalt	OpenVINS
EMH01	0.005118 ± 0.002989	0.011286 ± 0.006887
EMH02	0.004231 ± 0.002249	0.009638 ± 0.004916
EMH03	0.011682 ± 0.007597	0.020609 ± 0.011800
EMH04	0.012664 ± 0.008200	171.859600 ± 105.829991
EMH05	0.010716 ± 0.006293	0.020645 ± 0.012218
EV101	0.013046 ± 0.006272	0.014264 ± 0.006360
EV102	0.011127 ± 0.004774	0.015966 ± 0.009288
EV103	0.012287 ± 0.006093	0.014620 ± 0.007033
EV201	0.003423 ± 0.001966	0.008418 ± 0.006396
EV202	0.007064 ± 0.003854	0.009062 ± 0.004893
EV203	0.026494 ± 0.021414	0.024541 ± 0.016509 *
[AVG]	0.010519 ± 0.006350	0.014363 ± 0.008011

*: The original EV203 dataset has missing frames, it was modified to only use valid stereo frames for OpenVINS to improve scores.

An interesting example of trajectories with the two systems on EV101, notice how the loop closures cause abrupt corrections.