Windows when using pmap, PicklingError: Can't pickle <function create_objective_function.<locals>.f

[xgdgsc]

pars, details, _ = optunity.minimize(f, num_evals=100, pmap=optunity.parallel.create_pmap(2), x=[-5, 5], y=[-5, 5], solver_name='particle swarm')

when trying to use pmap in windows, it would raise exception:

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-24-af7e05b0af08> in <module>()
----> 1 pars, details, _ = optunity.minimize(f, num_evals=100, pmap=optunity.parallel.create_pmap(1), x=[-5, 5], y=[-5, 5], solver_name='particle swarm')

C:\Anaconda3\lib\site-packages\optunity\api.py in minimize(f, num_evals, solver_name, pmap, **kwargs)
    210     solver = make_solver(**suggestion)
    211     solution, details = optimize(solver, func, maximize=False, max_evals=num_evals,
--> 212                                  pmap=pmap)
    213     return solution, details, suggestion
    214 

C:\Anaconda3\lib\site-packages\optunity\api.py in optimize(solver, func, maximize, max_evals, pmap, decoder)
    243     time = timeit.default_timer()
    244     try:
--> 245         solution, report = solver.optimize(f, maximize, pmap=pmap)
    246     except fun.MaximumEvaluationsException:
    247         # early stopping because maximum number of evaluations is reached

C:\Anaconda3\lib\site-packages\optunity\solvers\ParticleSwarm.py in optimize(self, f, maximize, pmap)
    268 
    269         for g in range(self.num_generations):
--> 270             fitnesses = pmap(evaluate, list(map(self.particle2dict, pop)))
    271             for part, fitness in zip(pop, fitnesses):
    272                 part.fitness = fit * util.score(fitness)

C:\Anaconda3\lib\site-packages\optunity\parallel.py in pmap_bound(f, *args)
     93     def create_pmap(number_of_processes):
     94         def pmap_bound(f, *args):
---> 95             return pmap(f, *args, number_of_processes=number_of_processes)
     96         return pmap_bound
     97 

C:\Anaconda3\lib\site-packages\optunity\parallel.py in pmap(f, *args, **kwargs)
     76         for p in proc:
     77             p.daemon = True
---> 78             p.start()
     79 
     80         sent = [q_in.put((i, x)) for i, x in enumerate(zip(*args))]

C:\Anaconda3\lib\multiprocessing\process.py in start(self)
    103                'daemonic processes are not allowed to have children'
    104         _cleanup()
--> 105         self._popen = self._Popen(self)
    106         self._sentinel = self._popen.sentinel
    107         _children.add(self)

C:\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    210     @staticmethod
    211     def _Popen(process_obj):
--> 212         return _default_context.get_context().Process._Popen(process_obj)
    213 
    214 class DefaultContext(BaseContext):

C:\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    311         def _Popen(process_obj):
    312             from .popen_spawn_win32 import Popen
--> 313             return Popen(process_obj)
    314 
    315     class SpawnContext(BaseContext):

C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj)
     64             try:
     65                 reduction.dump(prep_data, to_child)
---> 66                 reduction.dump(process_obj, to_child)
     67             finally:
     68                 context.set_spawning_popen(None)

C:\Anaconda3\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
     57 def dump(obj, file, protocol=None):
     58     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 59     ForkingPickler(file, protocol).dump(obj)
     60 
     61 #

PicklingError: Can't pickle <function create_objective_function.<locals>.f at 0x0000000007BFBD08>: it's not the same object as __main__.f

On linux it is fine. Is it a limitation of windows version of multiprocessing that cannot be avoided?

[claesenm]

This appears to be a serialization problem, specific to Windows. It's surprising that this occurs, since we only use standard libraries for all these tasks, so in principle this should work on all Python platforms. Which Python version are you currently using? It should work for version 2.7 or higher, though admittedly we've never actually tested parallelization on Windows.

We could probably fix this by using some drop-in replacement of pickle (such as cloudpickle or dill), but I'd prefer not to add dependencies.

[xgdgsc]

I have anaconda python 3.4. http://stackoverflow.com/a/21345423/1136027 maybe helpful. Anyway, using linux is fine for me.

[ophiry]

I believe the issue is with serializing closures. In other cases replacing closures with classes with a call method solved the problem

[dmenig]

Hi. I'm experiencing the same issue. Would you care to exlain "replacing closures with classes with a call method solved the problem" ? I would very much like to be able to run optunity on my PC

[ophiry]

it appears that pickling funtions is problematic, so instead of defining a function: def f (): ....

use a callable class: class f_class: def call(): ....

and then p = f_class()

[SVasilev]

Hello, I ran into the same problem. @ophiry can you provide a full code snippet, because I cannot manage to implement the snippet you provided.

Also, @claesenm , will there be a fix for this. If not, it might be a good idea to add some section in the documentation about parallelization limitations in Windows. My operating system: Windows 10, 64bit