search

cloudpipe / cloudpickle

Extended pickling support for Python objects
Other
1.66k stars 167 forks source link

Can't pickle/unpickle Cython cdef classes #186

Open thomaspouncy opened 6 years ago

thomaspouncy commented 6 years ago

I'm not sure if this is a known limitations of cloud pickle that I was unaware of, but I haven't been able to get cloud pickle to pickle/unpickle instances of very basic cython-compiled classes.

If I have a TestClass.pyx file with the following content:

class TestClass(object):
    pass

, then compile that to a TestClass.so file with cython and run the following:

import cloudpickle

from TestClass import TestClass

pickled_obj = cloudpickle.dumps(TestClass())
print cloudpickle.loads(pickled_obj)

, everything works just fine. However, if I change TestClass to a cdef class:

cdef class TestClass(object):
    pass

, and recompile, I get the following error

Traceback (most recent call last):
  File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 6, in <module>
    print cloudpickle.loads(pickled_obj)
  File ".../lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File ".../lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File ".../lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
AttributeError: 'NoneType' object has no attribute '__pyx_unpickle_TestClass'

If I add any methods to this cdef class, like so:

cdef class TestClass(object):
    cdef void my_fxn(self):
        print "hello, world!"

, then I get the following error:

Traceback (most recent call last):
  File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 5, in <module>
    pickled_obj = cloudpickle.dumps(TestClass())
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
    cp.dump(obj)
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
    return Pickler.dump(self, obj)
  File ".../lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File ".../lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File ".../lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File ".../lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File ".../lib/python2.7/pickle.py", line 554, in save_tuple
    save(element)
  File ".../tpounce/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 654, in save_global
    return self.save_dynamic_class(obj)
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 501, in save_dynamic_class
    save(clsdict)
  File ".../lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File ".../lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File ".../lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File ".../lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
TypeError: can't pickle PyCapsule objects

Finally, if I add an init method to my cdef class:

cdef class TestClass(object):
    def __init__(self):
        pass

, I get this error:

Traceback (most recent call last):
  File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 5, in <module>
    pickled_obj = cloudpickle.dumps(TestClass())
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
    cp.dump(obj)
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
    return Pickler.dump(self, obj)
  File ".../lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File ".../lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File ".../lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File ".../lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File ".../lib/python2.7/pickle.py", line 554, in save_tuple
    save(element)
  File ".../lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 654, in save_global
    return self.save_dynamic_class(obj)
  File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 501, in save_dynamic_class
    save(clsdict)
  File ".../lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File ".../lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File ".../lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File ".../lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
TypeError: can't pickle wrapper_descriptor objects

Are all of these just known limitations of cloudpickle, or should this be working? I am using: cloudpickle==0.5.3 cython==0.28.3 python==2.7.13

Thanks!

ogrisel commented 6 years ago

Are cdef classes picklable by default with the pickle module from the standard library?

If not there is no way cloudpickle can do it automatically for you. I guess you need to implement a a __reduce__ method or the __getstate__ / __setstate_ method pair.