Benchmark idea: number of comparisons

[carlopi]

Benchmarks seems mostly to deal with types that are obvious to compare, I think it would be insightful (and coming mostly for free) to do benchmarks with number of comparison needed as metric (instead of time).

The idea would be still sorting integers or doubles, but passing a custom comparator that does counter++ and then returns the result. This would allow to give some insight regarding sorting costly-to-compare types.

[Morwenn]

Counting comparisons is pretty much what counting_adapter is for today, so the tool is already there. I do agree that such a benchmark is a good idea in theory (and probably in the scope of #128). Though if I add more I will soon hit a human issue: I've only got so much motivation to maintain benchmarks whenever there is a new release, and adding more benchmarks means more work unless I automate the process somehow.

[carlopi]

Nice infrastructure is already there, I will experiment with it.

I get the pain with benchmarking, I don't see any easy solution.

Feel free to close/merge the issue then.

[Morwenn]

The thing with benchmarking sorting algorithms is that if people need it, it means that they generally have rather specific ideas in mind and they're likely going to have to do it themselves anyway, so the best the library can do is to provide some tools. The available benchmarks are mostly rough guides to start somewhere.

Having good benchmarks would probably be a project of its own 😄

I'm going to close the issue then.