Problems with understanding the ouput of RPE

[UserUsingGit]

Hello, I am not quite understanding my results of APE and RPE. I am using Cartographer (only local SLAM) on the KITTI dataset and I am comparing the resulting trajectories of two data configurations. The first one is the original unmodified KITTI data (in bagfile format) and the second one is the same data, just with each frame having a shuffled order of points within the point cloud (which should actually not affect the algorithm). Surprisingly, I am getting different trajectories in the end. Why that is the case is probably a Cartographer issue and shall not be the topic here.

I plotted both trajectories using evo_traj: Here you can see that local SLAM is considerably less accurate using the unsorted_points as input. For this it makes sense that the absolute trajectory error (APE) is larger for the unsorted_points. But also the relative pose error (RPE) should differ (at least a little bit) in both configurations. I am using the default configurations for calculating and plotting the APE, for RPE I am using the angular error (delta = 1, delta_unit = frames). Left plots: sorted_points, right plots: unsorted points, upper plots: RPE, lower plots: APE

• the resulting errors of APE and RPE mapped onto the trajectory
• the respective error plots
• the respective terminal outputs (mean, rmse, ...) The large APE is clear to me as you can see the difference in the trajectory plots. In my understanding, this should mean a (at least slightly) larger (or different) RPE too. What I do not understand is the following: If I compare the ground truth with itself (giving two times ground truth as input), there should be no errors (or errors close to zero), which is roughly the case here: The trajectory of the sorted_points data is clearly closer to the ground truth data than it is to the unsorted_points data. Looking at the prints in the terminal, the mean error of the sorted_points is way closer to the mean error of the unsorted_points than to the ground truth data. This makes no sense to me looking at the trajectories, but I am sure I misundestood something.

Also, does a mean error of 1.03 (using the angular error per frame) means that averagely the rotation in every frame differs around 1 degree from the ground truth data? This seems very large to me. I hope someone can tell me what I am doing wrong here. Thanks alot in advance!

[ssssjiang]

嗷，我觉得1.03不是代表每帧相差一度吧，这里是rpe的计算方法https://github.com/MichaelGrupp/evo/blob/master/notebooks/metrics.py_API_Documentation.ipynb

[ssssjiang]

但是我猜evo_ape tum groundtruth.tum .txt test.txt -va --plot -r angle_deg 里（-r angle_deg）是代表计算每帧和真实数据相差多少度

[MichaelGrupp]

If I compare the ground truth with itself (giving two times ground truth as input), there should be no errors (or errors close to zero)

The error is strangely high when comparing the KITTI 00 groundtruth with itself:

# in evo/test/data
evo_rpe kitti KITTI_00_gt.txt KITTI_00_gt.txt -r angle_deg

I have no clue why this is the case at the moment. But with other trajectories the error is much smaller, also when they're converted to the KITTI format, for example:

# in evo/test/data
evo_rpe tum fr2_desk_groundtruth.txt fr2_desk_groundtruth.txt  -r angle_deg

evo_traj tum fr2_desk_groundtruth.txt --save_as_kitti

evo_rpe kitti fr2_desk_groundtruth.kitti fr2_desk_groundtruth.kitti  -r angle_deg

Here the max error is only 0.000005 degrees, which is what I would expect given small numerical errors.

---

Also, does a mean error of 1.03 (using the angular error per frame) means that averagely the rotation in every frame differs around 1 degree from the ground truth data?

You can have a look at the Jupyter notebook that @ssssjiang linked to see how it's calculated.

[UserUsingGit]

Thanks alot for the answers! I just realized that I did not include the commands that I used, so here is some additional information. For the RPE and APE I used the following commands:

evo_rpe kitti KITTI_sequence_00_groundtruth.txt sorted_points.txt --pose_relation angle_deg --delta 1 --delta_unit f --plot --plot_mode xz
evo_ape kitti KITTI_sequence_00_groundtruth.txt sorted_points.txt -va --plot --plot_mode xz

For unsorted_points.txt I used the exact same commands. I looked at the Jupyter notebook and cannot really see what i am doing wrong. So in the RPE, I specified the metric as angle in degrees per one frame. I did not specify it to run using --all_pairs since it would be the same results (delta = 1 frame is the smallest unit anyway). So as I understand, the mean is calculated by adding up all (4541) RPE values and dividing by this amount. So the mean in this case gives avarage the error angle per frame between the estimation and the reference, which would be 1.03 degrees (according to the results). But if I have an error of one degree per frame, how can the trajectory be even close to ground truth? Can it be the input data or am I missing something obvious?

[MichaelGrupp]

Sorry for the late reply. As mentioned in my previous comment, this strange behavior with the KITTI angular RPE needs some investigation. It's tracked here in a clean new issue: #216