search

cloudpipe / cloudpickle

Extended pickling support for Python objects
Other
1.64k stars 167 forks source link

pypy 3.8 fails to import cloudpickle #455

Closed dand-oss closed 1 year ago

dand-oss commented 2 years ago

pypy 3.8 release 7.3.6 does not find c extension _pickle

# python
Python 3.8.12 (279d80ac2079, Oct 17 2021, 05:25:30)
[PyPy 7.3.6 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import cloudpickle
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/pypy/lib/pypy3.8/site-packages/cloudpickle/__init__.py", line 4, in <module>
    from cloudpickle.cloudpickle import *  # noqa
  File "/pypy/lib/pypy3.8/site-packages/cloudpickle/cloudpickle.py", line 57, in <module>
    from .compat import pickle
  File "/pypy/lib/pypy3.8/site-packages/cloudpickle/compat.py", line 13, in <module>
    from _pickle import Pickler  # noqa: F401
ModuleNotFoundError: No module named '_pickle'
>>>>
p-fernandes commented 2 years ago

I would advise to drop the import of _pickle now.

bartbroere commented 2 years ago

Thanks for fixing this! Any idea when it might be included in a release?

(And for anyone googling it, symlinking pickle.py to _pickle.py will do the trick temporarily: 

ln -s /opt/pypy/lib/pypy3.8/pickle.py /opt/pypy/lib/pypy3.8/_pickle.py

)

schettino72 commented 2 years ago

@bartbroere really? just symlinking fixed the problem?

That did not work for me.

bartbroere commented 2 years ago

@bartbroere really? just symlinking fixed the problem?

That did not work for me.

Maybe I should have tested it a bit more, but we hit this part of cloudpickle through scikit-learn and joblib. When importing a metric from sklearn.metrics PyPy failed on the missing _pickle module. Since our code didn't actually use joblib or cloudpickle, the main thing it fixed was the errors on import of our metric.

Because joblib includes a copy of cloudpickle just pip installing the master here wasn't an option. For your use case, perhaps pip installing the current master is a better idea, since it's already fixed, just not yet released.

lesteve commented 2 years ago

I think this can be closed, since it has been fixed by https://github.com/cloudpipe/cloudpickle/pull/469.

mattip commented 2 years ago

Also #480 added pypy3.9 to the CI for testing, which is passing.

jakirkham commented 1 year ago

Thanks all! 🙏

Closing as resolved