search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
254 stars 85 forks source link

median budget cross goes astray with many repetitions #2287

Open nikohansen opened 1 month ago

nikohansen commented 1 month ago

See #2258 for considerations what to display. Otherwise, taking the median of the median of the median... is always less prone to be dominated by many repetitions of a single instance (or function). It would not work for f7 itself which is probably a good thing?

When making instance repetitions, see e.g. #2208, until a budget is exhausted, some algorithms will make very many repetitions on the step-ellipsoid, because they immediately stop on a plateau. As a consequence, the median budget over all trials from different problems becomes very low and is somewhat quite misleading as it gives more weight to the functions with a large number of trial repetitions.

A possible fix would be to take the median of the median of the problem instance budgets instead of the median of all problem instances.

nikohansen commented 1 week ago

Commit 3882ef6140 puts the crosses to the median of max(median(RT_u), median(RTs)) where median(RT*) is the runtime median over all (15+) instances of a DataSet, u=unsuccessful and s=successful subset.