search

cloudpipe / cloudpickle

Extended pickling support for Python objects
Other
1.63k stars 167 forks source link

Can't pickle functions that use the `colour-science` library #523

Open hotplot opened 10 months ago

hotplot commented 10 months ago

When trying to cloudpickle a function that references the colour-science library, the following exception is raised:

Traceback (most recent call last):
  File "/home/hrrsxb/aip/x/test.py", line 10, in <module>
    y = cloudpickle.loads(x)
  File "/home/hrrsxb/.cache/pypoetry/virtualenvs/x-Z-zmud14-py3.10/lib/python3.10/site-packages/colour/__init__.py", line 911, in __getattr__
    return super().__getattr__(attribute)
  File "/home/hrrsxb/.cache/pypoetry/virtualenvs/x-Z-zmud14-py3.10/lib/python3.10/site-packages/colour/utilities/deprecation.py", line 361, in __getattr__
    change = self._changes.get(attribute)

    ...

  File "/home/hrrsxb/.cache/pypoetry/virtualenvs/x-Z-zmud14-py3.10/lib/python3.10/site-packages/colour/__init__.py", line 911, in __getattr__
    return super().__getattr__(attribute)
  File "/home/hrrsxb/.cache/pypoetry/virtualenvs/x-Z-zmud14-py3.10/lib/python3.10/site-packages/colour/utilities/deprecation.py", line 361, in __getattr__
    change = self._changes.get(attribute)
RecursionError: maximum recursion depth exceeded

Minimal code to reproduce:

import colour
import cloudpickle

def foo():
    print(colour.__version__)

x = cloudpickle.dumps(foo)
y = cloudpickle.loads(x)

The issue does not occur when using standard pickle

Versions used are: python: 3.10.12 cloudpickle: 2.2.1 and 3.0.0 both tested colour: 0.4.3

ogrisel commented 10 months ago

The colour module is interactively redefined at import time here:

https://github.com/colour-science/colour/blob/v0.4.3/colour/__init__.py#L967

the fake module itself is define here:

https://github.com/colour-science/colour/blob/v0.4.3/colour/__init__.py#L905

and has a generic __getattr__ to lookup attributes + some class level attributes that are defined a-posteriori when first importing the real module.

I suspect that since it's a fake module, cloudpickle is also trying to pickle it by value instead of pickling it by reference. And the way it's designed it cannot be pickled by value.

Maybe cloudpickle's heuristic to detect what module should be pickled by value could be improved.

Alternatively, colour itself might be fixed or improved to make it safe to pickle its fake module by value, for instance by defining the class attributes in the class definition itself rather than by setting them a posteriori.