uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

exception when use pathos with deap on windows for multiprocess #221

Open liningbo opened 2 years ago

liningbo commented 2 years ago

the following is the exception: Process SpawnPoolWorker-6: Traceback (most recent call last): File "F:\ProgramData\Anaconda3\lib\site-packages\multiprocess\process.py", line 315, in _bootstrap self.run() File "F:\ProgramData\Anaconda3\lib\site-packages\multiprocess\process.py", line 108, in run self._target(*self._args, self._kwargs) File "F:\ProgramData\Anaconda3\lib\site-packages\multiprocess\pool.py", line 114, in worker task = get() File "F:\ProgramData\Anaconda3\lib\site-packages\multiprocess\queues.py", line 361, in get return _ForkingPickler.loads(res) File "F:\ProgramData\Anaconda3\lib\site-packages\dill_dill.py", line 327, in loads return load(file, ignore, kwds) File "F:\ProgramData\Anaconda3\lib\site-packages\dill_dill.py", line 313, in load return Unpickler(file, ignore=ignore, **kwds).load() File "F:\ProgramData\Anaconda3\lib\site-packages\dill_dill.py", line 525, in load obj = StockUnpickler.load(self) File "F:\ProgramData\Anaconda3\lib\site-packages\dill_dill.py", line 515, in find_class return StockUnpickler.find_class(self, module, name) AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from 'F:\ProgramData\Anaconda3\lib\site-packages\deap\creator.py'>

if the following code:

creator.create("FitnessMin", base.Fitness, weights=(-1.0,)) creator.create("Individual", list, fitness=creator.FitnessMin)

is in 'if name== "main":'

the exception will raise on windows but not raise on mac.

on the stackoverflow, some gay has met the same problem : https://stackoverflow.com/questions/61902235/unable-to-speed-up-python-deap-with-multiprocessing

on windows ,it can be solved by put the code 'creator.create' outside of 'if name== "main":'

my question is:

  1. why the excetion happened on windows with pathos
  2. can pathos avoid the exception on windows when the code 'creator.create' is in the 'if name== "main":'

Thanks