about evaluation using evo

[1216621137]

I want to evaluate results output by vins using this dataset. I use the evo tool to do this work, but I am confused about the alignment between the groundtruth and the vins output trajectory. here are my evaluation results using different alignment options:

note that "circle_vins.bag" contains the trajectory output by vins and the gt in dataset. I have got different rmse results and I wanna know which one is corresponding to the results presented in the paper? To be specific, these are some qustions: 1) Has the groundtruth in dataset been aligned? 2) Which one of the above results correspoing to the results presented in the paper.

[zinuok]

Dear @1216621137,

Thanks for your interest in our dataset.

A1. The ground truth pose was not aligned because most recent tools, such as evo or uzh-rpg have a function to align the estimated trajectory and its ground truth.

A2. The last one with "--align_origin" is the correct one shown in our paper. As described in Table IV (please see the caption in the table), all trajectory errors were evaluated with the origin alignment method. If you use the "-a" flag, the trajectories are aligned with the Umeyama method, where the alignment is performed using SVD (for least-squares estimation) so that the trajectories overlap as much as possible, and ATE errors can be reduced.

Thank you Jinwoo Jeon

[1216621137]

Thank you! That helps a lot. Here is another question about the ground truth. I notice that the first one (without any alignment options) on the above attached picture also shows a not bad result. However, the ground truth is supposed to be expressed in a global frame which is different from the vio frame. Has any preprocessing done to the groundtruth? I have done a similar work using a motion capture system to record the groundtruth. When I directly choose the "--align_origin" to align the two trajectories, these two trajectories are only aligned at the origin, and the rest part is totally in a mess. So I wonder if there is some preprocessing done for the groundtruth in your dataset.

[zinuok]

Yes, you mentioned the exact point. As you correctly mentioned, the VIO frame is actually a local frame where the origin is defined as the starting point (0, 0, 0).

Therefore, we multiplied the inverse transformation matrix based on the first acquired ground truth pose (from the Motion capture system) to equate the coordinate system defined in motion capture with the VIO frame. This was done with a separate ROS node. That's why the results aren't bad, even if you didn't do any alignment.

[1216621137]

Thanks a lot ! And thanks for your brilliant work. I'll try to correct the origin before using the evaluation tool.

[1216621137]

Another question: As for VINS-Fusion, which topic is applied to evaluate the ATE? /vins_estimator/keyframe_pose, /vins_estimator/odometry or //vins_estimator/imu_propagate?

[zinuok]

We used '/vins_estimator/odometry'. Please note that '/vins_estimator/imu_propagate' has low accuracy, which is unsuitable for fair performance comparisons

[1216621137]

Thank you. May I ask you some more questions? I noticed that in many papers about vio, the authors rarely mentioned which method（--align_origin or -a, etc.） they used for coordinate alignment, and many datasets seem to have no similar preprocessing for ground truth as you did. So I wonder if it is ok to just use a consistent alignment method, regardless of which one.

[zinuok]

The alignment method does not affect the 'RPE' metric, which is common for the odometry evaluation. However, the 'ATE' metric can be changed according to the alignment method. It's up to you, as long as you are using a consistent method.