search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
262 stars 87 forks source link

Timing of the example experiment #484

Closed nikohansen closed 8 years ago

nikohansen commented 8 years ago

just to have a rough idea, on a 2014 MacBook Pro: 

            2014 MacBook Pro  | 2009 Dell Win | 2015 HP Win
|  time[s]  startup   run     |               |          
|  C:         0        5      |   60          |   55
|  Java:      0        5      |               | 
|  Matlab:                    |   50          |
|  Octave:    1       91      |               |  125
|  Python:    2        5      |               |

The startup time on Python is to a large part a consequence of touching each problem once to collect some information before to start the benchmarking. Feel free to provide some data for the missing entry.

EDIT: updated with below numbers.

brockho commented 8 years ago

My numbers are not fully comparable, because on one machine, I have installed only Matlab (Windows XP on an old 2009 Dell notebook) on the other only Octave (Windows 7 on a 2015 HP notebook). For both tests, the C code has been compiled for 32bit.

Octave runs in 123s...133s, Matlab in 48s...53s (numbers of 3 runs each).

nikohansen commented 8 years ago

If you add the data with, say, C, we have a common point of comparison.

brockho commented 8 years ago

Surprisingly, the C experiment also takes around 55 seconds (on the new Windows 7 notebook) and around 60 seconds (for the old Windows XP notebook). Almost the same numbers hold for java and python on the Windows 7 notebook.

brockho commented 8 years ago

Can this issue be closed? Or do we want to decrease the number of function evaluations in the example experiments further?

nikohansen commented 8 years ago

If 100+ seconds is actually a standard scenario, I am in favor of decreasing the number of evaluations.

brockho commented 8 years ago

The new rewrite of the Matlab wrapper (incl. the new example experiment with restarts) made the exampleexperiment.m under Octave a bit quicker. It takes now about 109 seconds for the biobjective experiment with a budget of 2DIM. Reducing the budget to DIM/2 reduces the time down to about 73 seconds in Octave. The same in C still takes around 55 seconds for a 2DIM budget and about 50 seconds for a DIM/2 budget.

nikohansen commented 8 years ago

Strange, what is going on here?