Map quality metrics

[YoshuaNava]

Is your feature request related to a problem? Please describe. SLAM systems are usually able to output an estimated pose and a map. Evo currently provides support for pose accuracy metrics, such as APE and RPE, and nice methods to plot and analyze trajectories. However, it can not be used to compare maps.

Describe the solution you'd like Introduce map quality metrics, for example:

• Reconstruction error [1].
• Mean map entropy and mean plane variance [2].

[1] SLAMBench2 [2] point_cloud_evaluation_tool

Describe alternatives you've considered To run existing open source packages on the output of my SLAM algorithms, but there is no clear workflow defined for them, input/output types vary, and some implementations are tightly coupled with unrelated things.

Additional context This would be a very nice feature to integrate into evo. However, I know it might require multiple changes here and there. I'm opening this issue considering that and hoping we can bound some ideas (if you agree with my feature request)

There are important aspects to consider, like:

• Types of maps (2D, 3D) and representation (dense point clouds, voxels, sparse keyframes, etc)
• Input formats for maps.
• Internal representation of maps that can be generic enough for different map types.
• Keeping dependencies of evo to a minimum.
• Which metrics actually make sense to have.

[MichaelGrupp]

I would recommend to take a look at software like Cloudcompare that allows to do comparisons between point clouds & meshes and much more: https://www.danielgm.net/cc/

As you mentioned, it is hard to define what a "map" contains across different SLAM implementations. Trajectories and point clouds or meshes are things that are comparable though. This package focuses on trajectories and I don't think you can add point cloud support without major changes. But luckily there's already CloudCompare, which is an excellent tool for that purpose.

[YoshuaNava]

I would recommend to take a look at software like Cloudcompare that allows to do comparisons between point clouds & meshes and much more

I'm aware of cloud compare, and even though it is quite cool, robust and user friendly, I consider that there is much more than you could evaluate for SLAM.

For example, in SLAM (vision, point cloud-based, etc), the concept of landmarks is quite relevant, and there is a correlation between trajectories and landmarks that is not exploited for assessing the quality of maps atm. Furthermore, I believe that CloudCompare does a fine job at comparing point clouds / meshes based on their point distance, but this is not enough. SLAM landmarks usually have other measures of compatibility that are not taken into account.

[MichaelGrupp]

SLAM landmarks usually have other measures of compatibility that are not taken into account

Do you have any specific algorithms in mind that could be implemented? Ideally with some example input data.