search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

ProcessPool in 3.x fails with PicklingError for dynamically created classes #156

Closed benjaminpatrickevans closed 5 years ago

benjaminpatrickevans commented 5 years ago

Hi there, I'm having some issues getting the ProcessPool to work with custom defined new type objects. A minimal example

import numpy as np
import dill
import pathos

print(pathos.__version__) # '0.2.2.1'
print(dill.__version__) # 0.2.8.2

custom_type = type("ABC", (int, ), {})
instance = custom_type(5)
list_of_instances = [instance] * 10

map(np.add, list_of_instances, [1] * len(list_of_instances)) # Works

dill.copy(list_of_instances) # Works

pool = pathos.multiprocessing.ProcessPool(1)
pool.map(np.add, list_of_instances, [1] * len(list_of_instances)) # Fails, _pickle.PicklingError: Can't pickle <class '__main__.ABC'>: it's not found as __main__.ABC

# And trying another Pool for good luck
pool = pathos.pools.ProcessPool(1)
pool.map(np.add, list_of_instances, [1] * len(list_of_instances)) # Also Fails, _pickle.PicklingError: Can't pickle <class '__main__.ABC'>: it's not found as __main__.ABC

This fails with

_pickle.PicklingError: Can't pickle <class 'main.ABC'>: it's not found as main.ABC

Full stack of the error

_pickle.PicklingError: Can't pickle <class 'main.ABC'>: it's not found as main.ABC Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.7/site-packages/pathos/multiprocessing.py", line 137, in map return _pool.map(star(f), zip(args)) # chunksize File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get raise self._value File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks put(task) File "/usr/local/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/local/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol).dump(obj) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump self.save(obj) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save self.save_reduce(obj=obj, rv) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 633, in save_reduce save(cls) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 1293, in save_type StockPickler.save_global(pickler, obj) File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global (obj, module_name, name)) from None _pickle.PicklingError: Can't pickle <class 'main.ABC'>: it's not found as main.ABC

If instead custom_type is declared as

class custom_type(int):
    pass

A new error arrises with pool.map

File "/usr/local/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, self._kwargs) File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 110, in worker task = get() File "/usr/local/lib/python3.7/site-packages/multiprocess/queues.py", line 357, in get return _ForkingPickler.loads(res) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 316, in loads return load(file, ignore) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 304, in load obj = pik.load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 465, in find_class return StockUnpickler.find_class(self, module, name) AttributeError: Can't get attribute 'custom_type' on <module 'main' (built-in)>**

Yet dill.copy still works perfectly fine in both cases

Any idea what could be going on here? Thanks!

mmckerns commented 5 years ago

I'm not able to reproduce your error (admittedly, with a slightly newer version of pathos).

>>> import numpy as np
>>> import dill
>>> import pathos
>>> 
>>> print(pathos.__version__) # '0.2.2.1'
0.2.3.dev0
>>> print(dill.__version__) # 0.2.8.2
0.2.8.2
>>> 
>>> custom_type = type("ABC", (int, ), {})
>>> instance = custom_type(5)
>>> list_of_instances = [instance] * 10
>>> 
>>> map(np.add, list_of_instances, [1] * len(list_of_instances)) # Works
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> 
>>> dill.copy(list_of_instances)
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> 
>>> dill.check(list_of_instances)
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> pool = pathos.multiprocessing.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>>
mmckerns commented 5 years ago

Oh, wait... I was using 2.7. I see... it does fail for 3.7.

mmckerns commented 5 years ago 
>>> dill.detect.trace(True)
>>> dill.check(list_of_instances)
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x10c777048>
# F2
T1: <class 'type'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
T1: <class 'int'>
# T1
D2: <dict object at 0x10c9252d0>
# D2
# T2
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>>

And the failure case...

>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 ...
_pickle.PicklingError: Can't pickle <class '__main__.ABC'>: it's not found as __main__.ABC
mmckerns commented 5 years ago

In 2.7:

>>> dill.detect.trace(True)
>>> dill.check(list_of_instances)
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
F2: <function _load_type at 0x105cb0938>
# F2
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105da1e88>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>>

and...

>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <type 'code'>
F2: <function _load_type at 0x105cb0938>
# F2
# T1
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105dd0e88>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd0d70>
# D2
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <type 'code'>
F2: <function _load_type at 0x105cb0938>
# F2
# T1
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
B2: <built-in function scalar>
T1: <type 'int'>
# T1
# B2
D2: <dict object at 0x105dd07f8>
T4: <type 'numpy.dtype'>
# T4
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd06e0>
# D2
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
B2: <built-in function scalar>
T1: <type 'code'>
# B2
F2: <function _load_type at 0x105cb0938>
# F2
T4: <type 'numpy.dtype'>
# T1
# T4
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105dd0c58>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd0d70>
# D2
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <type 'code'>
F2: <function _load_type at 0x105cb0938>
B2: <built-in function scalar>
# F2
# B2
# T1
T4: <type 'numpy.dtype'>
# T4
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105dd0e88>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd06e0>
# D2
B2: <built-in function scalar>
# B2
T4: <type 'numpy.dtype'>
# T4
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>>
mmckerns commented 5 years ago

Hmm... the primary difference seems to be that for 3.7, we have T5: <class '__main__.ABC'> which means that the dynamically created class ABC is being passed to pickle (and failing) -- while in 2.7, we have T2: <class '__main__.ABC'>, which uses the dill function _dict_from_dictproxy. Notice that directly using dill also utilizes T2 in both 3.7 and 2.7.

mmckerns commented 5 years ago

Ok, so there's a workaround for now, since this is falling back to pickle, and I have to figure out why it is for 3.7. Python's pickler requires that the name of the dynamically created class is the same as the name of the instance:

>>> import numpy as np
>>> import dill
>>> import pathos
>>> 
>>> print(pathos.__version__) # '0.2.2.1'
0.2.3.dev0
>>> print(dill.__version__) # 0.2.8.2
0.2.9.dev0
>>> 
>>> custom_type = type("custom_type", (int, ), {})
>>> instance = custom_type(5)
>>> list_of_instances = [instance] * 10
>>> 
>>> map(np.add, list_of_instances, [1] * len(list_of_instances)) # Works
<map object at 0x10fd9ab00>
>>> 
>>> dill.check(list_of_instances) # Works
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> 
>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>>
mmckerns commented 5 years ago

This is fixed by uqfoundation/dill@3b043d67c6e860a5b558bfafa9569a9ae6d54700. Using the most recent version of dill, your original test code now works for 3.7, etc.

Please reopen this ticket if you are still experiencing an error.

rakhmaevao commented 2 years ago

Hello. I have same error for:

Python: 3.9.7
Pathos: 0.2.8
dill: 0.3.4

Example code:

from abc import abstractmethod, ABC
from multiprocessing import Pool
from pathos.pools import ProcessPool
import pathos
import dill
import platform

print(f'Python: {platform.python_version()}')
print(f'Pathos: {pathos.__version__}')
print(f'dill: {dill.__version__}')

class SomeAbstractClass(ABC):
    @abstractmethod
    def some_methode(self):
        pass

class SomeConreteClass(SomeAbstractClass):
    def __init__(self, blank=None):
        self.some_methode()

    def some_methode(self):
        print('something')

if __name__ == '__main__':
    with ProcessPool(5) as p:
        p.map(SomeConreteClass, [1, 2, 3])
Error output: ``` Python: 3.9.7 Pathos: 0.2.8 dill: 0.3.4 Traceback (most recent call last): File "C:\Users\rahma\Documents\detectionzone\выпаа.py", line 28, in p.map(SomeConreteClass, [1, 2, 3]) File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\pathos\multiprocessing.py", line 139, in map return _pool.map(star(f), zip(*args)) # chunksize File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\pool.py", line 771, in get raise self._value File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\pool.py", line 537, in _handle_tasks put(task) File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\connection.py", line 214, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\reduction.py", line 54, in dumps cls(buf, protocol, *args, **kwds).dump(obj) File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 498, in dump StockPickler.dump(self, obj) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 487, in dump self.save(obj) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1493, in save_function pickler.save_reduce(_create_function, (obj.__code__, File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce save(args) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1227, in save_cell pickler.save_reduce(_create_cell, (f,), obj=obj) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce save(args) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1422, in save_type pickler.save_reduce(_create_type, (type(obj), obj.__name__, File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce save(args) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1422, in save_type pickler.save_reduce(_create_type, (type(obj), obj.__name__, File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce save(args) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple save(element) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 997, in _batch_setitems save(v) File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 578, in save rv = reduce(self.proto) TypeError: cannot pickle '_abc._abc_data' object Process finished with exit code 1 ```
mmckerns commented 2 years ago

@rahmaevao: your issue is not the same as the original issue. Yours is a duplicate of https://github.com/uqfoundation/dill/issues/332.

Larleyt commented 2 years ago

The issue still persists for

Python==3.9
dill==0.3.4
pathos==0.2.8 
mlflow==1.24.0

Basically, in the __init__() of a class I'm trying to pickle there's an mlflow.pyfunc.load_model() function executing. I have a suspicion dill cannot pickle a loaded model to split it across processes correctly. Am I right? Strangely, build-in multiprocessing package works ok with this case.

TracebackTraceback (most recent call last): File "project_name/feature_pipeline/main.py", line 7, in application.run() File "project_name/feature_pipeline/application.py", line 55, in run dataframe = feature_calculator.run() File "project_name/feature_pipeline/call_methods.py", line 271, in run self._call_method(method, field_candidates) File "project_name/feature_pipeline/call_methods.py", line 238, in _call_method for dataframe in pool.imap( File "project_name/.venv/lib/python3.9/site-packages/multiprocess/pool.py", line 870, in next raise value File "project_name/.venv/lib/python3.9/site-packages/multiprocess/pool.py", line 537, in _handle_tasks put(task) File "project_name/.venv/lib/python3.9/site-packages/multiprocess/connection.py", line 214, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "project_name/core-tss.ds/.venv/lib/python3.9/site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol, *args, **kwds).dump(obj) File "project_name/core-tss.ds/.venv/lib/python3.9/site-packages/dill/_dill.py", line 498, in dump StockPickler.dump(self, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 487, in dump self.save(obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce save(state) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 1002, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce save(state) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce save(state) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 687, in save_reduce save(cls) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1422, in save_type pickler.save_reduce(_create_type, (type(obj), obj.__name__, File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1493, in save_function pickler.save_reduce(_create_function, (obj.__code__, File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1227, in save_cell pickler.save_reduce(_create_cell, (f,), obj=obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1410, in save_type StockPickler.save_global(pickler, obj, name=name) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 1070, in save_global raise PicklingError( _pickle.PicklingError: Can't pickle : it's not found as __main__.ClusteringSystem
mmckerns commented 2 years ago

I have a suspicion dill cannot pickle a loaded model to split it across processes correctly. Am I right?

No. dill was build specifically to work across processes (see multiprocess and pathos).

It's a bit hard to determine what's going on with your code when you only posted the traceback (and not the code itself). However, it looks like somewhere down the traceback you are pickling a function, and it's then trying to pickle the global dict, and then there's a thing called ClusteringSystem which it expects to be defined in __main__ but it can't be found. You might try to use dill.settings['recurse'] = True.

However, it does not seem that your issue is the same as the original one, based on the above.