esa / pygmo2

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.
https://esa.github.io/pygmo2/
Mozilla Public License 2.0
414 stars 56 forks source link

speeding up ipyparallel evolutions [FEATURE] #156

Open dalbabur opened 4 months ago

dalbabur commented 4 months ago

Is your feature request related to a problem? Please describe. Sending the problem and population to every ipyparallel engine with every evaluation is fairly slow, specially when using many (>100) engines. If the problem does not change during the evolution, it should be enough to send it only once. Additionally, it may be worth having the option to read/write the population to disk, completely avoiding the scatter step.

Describe the solution you'd like I think I'm looking for a new bfe, that does not pickle everything every time, and that can use other ipyparallel views (such as BroadcastView). Would other aspects of pygmo have to change to enable this?