search

rainwoodman / sharedmem

A different flavor of multiprocessing in Python
GNU General Public License v3.0
81 stars 9 forks source link

MapReduce no longer working on Python 3.8 #24

Open ezwelty opened 3 years ago

ezwelty commented 3 years ago

The following script works for me under Python 3.7.7 (and earlier), but fails under Python 3.8. I suspect it has something to do with changes to the pickling by the underlying multiprocessing module. The script works in Python 3.8 if MapReduce is changed to MapReduceByThread.

import sharedmem

def f(i):
    return i

with sharedmem.MapReduce(np=2) as pool:
    print(pool.map(func=f, sequence=(1, 2)))

Here is the error in Python 3.8 (tested on both 3.8.2 and 3.8.5):

Traceback (most recent call last):
  File "/Users/Admin/Library/Caches/pypoetry/virtualenvs/glimpse-QRj6g5nk-py3.8/lib/python3.8/site-packages/sharedmem/sharedmem.py", line 735, in map
    pg.start()
  File "/Users/Admin/Library/Caches/pypoetry/virtualenvs/glimpse-QRj6g5nk-py3.8/lib/python3.8/site-packages/sharedmem/sharedmem.py", line 389, in start
    x.start()
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/Admin/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ProcessBackend.StorageFactory.<locals>.<lambda>'

Any thoughts on what may be causing this and if you suspect an easy or a hard fix to sharedmem?