search

numbbo / coco-visualize

Bare bones runtime profile plots from benchmark results.
Other
1 stars 0 forks source link

Ensure "balanced" results by algorithm #7

Open olafmersmann opened 2 months ago

olafmersmann commented 2 months ago

When aggregating over multiple problems / problem instances, we can get imbalanced results w.r.t. the number of runs of each algorithm.

Idea: Drop results and warn about excluded results.

nikohansen commented 2 months ago

If we display the number of repetitions next to the problem name, we could have for each problem a different repetition number given all algorithms have the same number of repetitions or we undersample algorithms that have more repetitions. The displayed repetition number would then be proportional to the weight of this problem in the graphs. It could be an option to either undersample to have the same repetitions over all algorithms and problems or only over all algorithms.