Evaluate odometry accuracy

[yuhang1008]

Hi there. I have modified the config files and able to run DM-VIO on my own dataset. The result is very nice and impressive.

For this ROS version, could someone give further guidance on how to evaluate the odometry? I noticed it is not suggested to use topic dmvio/metric_pose directly because the scale is not finalized. Then how should we modify the RosOutputWrapper.cpp?

Thanks in advance.

[lukasvst]

Great to hear that it works well on your dataset!

For evaluating the odometry you should take the entire unscaled trajectory (frame_tracked/pose) and scale it with one consistent scale (published as frame_tracked/scale). (You might also have to convert from camera to IMU frame). You will likely get the best result by using the last published scale to scale the trajectory. The outputted file resultScaled.txt does this, so it can also be used instead of ROS topics.

While this evaluation is not fully causal (see Okvis 2 paper for definition), many other SLAM systems are evaluated non-causally as well, and I'd argue that in this case it makes sense to evaluate the scale non-causally. E.g. for many robotics applications, one can use the non-scaled trajectory for planning and the scaled trajectory for control.

Alternatively you can use an earlier scale (e.g. a couple of seconds after initializing the VIO), and scale the trajectory with it, if you are interested in a more causal result. In that case it might make sense to set setting_scaleFixTH=0.007 which will fix the scale after some time. Then you can just use the fixed scale. But expect to get slightly worse accuracy with this, compared to the first option above.

[yuhang1008]

Great to hear that it works well on your dataset!

For evaluating the odometry you should take the entire unscaled trajectory (frame_tracked/pose) and scale it with one consistent scale (published as frame_tracked/scale). (You might also have to convert from camera to IMU frame). You will likely get the best result by using the last published scale to scale the trajectory. The outputted file resultScaled.txt does this, so it can also be used instead of ROS topics.

While this evaluation is not fully causal (see Okvis 2 paper for definition), many other SLAM systems are evaluated non-causally as well, and I'd argue that in this case it makes sense to evaluate the scale non-causally. E.g. for many robotics applications, one can use the non-scaled trajectory for planning and the scaled trajectory for control.

Alternatively you can use an earlier scale (e.g. a couple of seconds after initializing the VIO), and scale the trajectory with it, if you are interested in a more causal result. In that case it might make sense to set setting_scaleFixTH=0.007 which will fix the scale after some time. Then you can just use the fixed scale. But expect to get slightly worse accuracy with this, compared to the first option above.

You explained my problem very clearly. Thank you so much Lukas!