search

esa / pygmo2

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.
https://esa.github.io/pygmo2/
Mozilla Public License 2.0
442 stars 56 forks source link

Documentation of parallel processing of individual fitness function evaluations #13

Closed mike-slackenerny closed 3 years ago

mike-slackenerny commented 4 years ago

I have been tinkering with pygmo to automate the optimisation of a 3D structure in an electromagnetism problem. The evaluation of the fitness function can take up to 1h on an AWS cluster. In this type of problem, sequential computation of generations of more than a couple of individuals is out of the question. The batch parallel initialization helps, but it is of course just the first step.

I was wondering if any of the algorithms in the library could use this same parallel evaluation for subsequent generations, or whether it would be possible to launch individual evaluations in parallel through the existing mp/thread/ipyparallel based classes. As long as the calculation of the parameters of the individuals in a generation do not depend from the results of previous members of their generation, I guess this should work.

mike-slackenerny commented 4 years ago

OK, I joined the Gitter chat and saw some answers on this there. You need to use the set_bfe() method of the algorithm before calling the constructor for pygmo.algorithm

Something like this is working for me now

prob = pg.problem(my_problem())
ants = pg.gaco(gen=20,ker=24)
ants.set_bfe(pg.bfe())
algo = pg.algorithm(ants)
ils = pg.island(algo=algo,prob=prob,size=24,udi=pg.mp_island(), b=pg.default_bfe())
ils.evolve(n=10)

I am editing this issue to focus on documentation of these new capabilities: what algorithms support this parallelisation, example code, etc.

Thanks for the great work, BTW!

bluescarni commented 4 years ago

@mike-slackenerny Yes that is correct, the bfe machinery is used to implement both parallel initialisation and parallel fitness function evaluation in the algorithms that support it (currently NSGA2, GACO, MHACO, PSO_GEN and NSPSO, I believe).

We definitely need better tutorial-style documentation for this feature.

bluescarni commented 4 years ago

Moving the issue over to esa/pygmo2.

sertho commented 4 years ago

Hi Thank you for your amazing work PyGMO! I just published a question on stackexchange regarding this topic: https://stackoverflow.com/questions/60435885/pygmo-batch-fitness-evaluation

Would be nice if in the meantime somebody could give a short example as answer on that question?

bluescarni commented 4 years ago

I have replied on SO.