search

MichaelGrupp / evo

Python package for the evaluation of odometry and SLAM
https://michaelgrupp.github.io/evo/
GNU General Public License v3.0
3.33k stars 745 forks source link

Why is --all_pairs not the default? #592

Closed kax-slamcore closed 7 months ago

kax-slamcore commented 9 months ago

Hi Michael,

I note that you previously stated that you think that using --all_pairs produces more statistically robust results here.

Is there a reason why it's not the default behaviour? Other than execution time, is there a reason not to use it?

Many thanks for the tools and extensive community support!

MichaelGrupp commented 7 months ago

There is not a reason why you shouldn't use all_pairs in general. The default "non all pairs" method is simpler to understand, e.g. for visual analysis in plots. That was the reason why it was set as default.

Changing the default now would change the behavior of the user interface, so I don't want to do that.

kax-slamcore commented 7 months ago

Hi Michael,

Thanks for the response. I'm happy to close this :tada: .

However, perhaps you might be interested in an insight I've had on the matter since opening the issue:

There is a problem when using --all_pairs with -u m (i.e. setting the denominator to a distance) on a trajectory with a stationary section. In this case there will be many "pairs" that begin at that same stationary point (or very close to) and end at the same point (or very close to), the end point being the set distance along the reference trajectory. The error_array value for each of these "pairs" will be very similar.

The result is that this error_array value is over-represented in the sample. Whatever this value is, it really only occurs over a single span, however, it will be sampled as many times as there are poses in the stationary segment. This will alter the sample and change the reported mean, median, rmse, sse and stdev measures.

I have a feeling (and of course I have no way of knowing) that many if not most SLAM trajectories contain a stationary segment, likely at the beginning, so for me this is not an extreme corner case.

MichaelGrupp commented 7 months ago

True, this would be a disadvantage of the method where the default sequential RPE could give more balanced results. Or some other form of spatial subsampling.