search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.39k stars 89 forks source link

PicklingError: Can't pickle <type 'Environment'>: it's not found as __builtin__.Environment #109

Closed nmz787 closed 7 years ago

nmz787 commented 7 years ago

I was trying to run map on an instance method, from an instance method of the same object

  File "C:\Users\User\git\my_script.py", line 171, in do_it_mp
    xs = p.map(self.match_and_add_info, self.data_to_match)
  File "C:\Python27\lib\site-packages\pathos\multiprocessing.py", line 137, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 567, in get
    raise self._value
PicklingError: Can't pickle <type 'Environment'>: it's not found as __builtin__.Environment
    def do_it_mp(self):
        p = p = mp.ProcessingPool(4)
        xs = p.map(self.match_and_add_info, self.data_to_match)
mmckerns commented 7 years ago

The rather oblique raise self._value and then a PicklingError, when on Windows, is usually an indication that you don't have a full install of multiprocess. If you don't have a C compiler, you need one... then reinstall multiprocess.

If that doesn't fix it for you, then please post a reduced version of your code that I can try that reproduces your error.

dnminh11 commented 7 years ago

I'm having the same problem too. Can you name some specific C compilers? I tried MinGW and the problem hasn't been solved yet.

dnminh11 commented 7 years ago

Updating: Finally i've found the problem. I was using Python 2.7 so the code runs with error, but after i updated to Python 3.6, everything works fine.

mmckerns commented 7 years ago

@nmz787: is this still an issue for you? I don't have anything I can do here without code that can reproduce your error. @dnminh11: feel free to share code where you feel you may have been seeing this issue as well.

mmckerns commented 7 years ago

@dnminh11: one of the reasons that 3.6 probably works and 2.7 doesn't is that 3.6 doesn't need a C compiler for hardly any of the code, so I'm going to assert that this is the answer to this issue (for both you and @nmz787). Either of you please feel free to reopen this ticket if you discover differently.