Override default pickling behavior

[gpauloski]

I am trying to override the default pickling behavior of Proxy to instead pickle the factory. This works for lazy_object_proxy.cext.Proxy but not for lazy_object_proxy.slots.Proxy.

from lazy_object_proxy.slots import Proxy

class MyProxy(Proxy):
    def __reduce__(self):
        return MyProxy, (self.__factory__,)
    def __reduce_ex__(self, protocol):
        return MyProxy, (self.__factory__,)

Then, pickling an instance of MyProxy gives the following error:

Traceback (most recent call last):
  File "test.py", line 37, in <module>
    proxy_pkl = pkl.dumps(proxy_instance)
_pickle.PicklingError: Can't pickle <class 'MyProxy'>: import of module <property object at 0x7f42833a50b0> failed

The factory in this case is a callable, pickleable object that returns a large numpy array, but I do not want to include the large array in the pickling (as done with the default behavior).

Is there a workaround for pickling the slots version of a Proxy? I am not using the lazy_object_proxy.cext.Proxy because I have run into some strange behavior with common numpy functions.

[ionelmc]

Afaik the pickling should work ok for all the implementation. Perhaps there's some corner case I missed in the tests - please provide a reproducer.

On Tue, Apr 27, 2021, 22:45 Greg Pauloski @.***> wrote:

I am trying to override the default pickling behavior of Proxy to instead pickle the factory. This works for lazy_object_proxy.cext.Proxy but not for lazy_object_proxy.slots.Proxy.

from lazy_object_proxy.slots import Proxy class MyProxy(Proxy): def reduce(self): return MyProxy, (self.factory,) def __reduce_ex(self, protocol): return MyProxy, (self.factory__,)

Then, pickling an instance of MyProxy gives the following error:

Traceback (most recent call last): File "test.py", line 37, in proxy_pkl = pkl.dumps(proxy_instance) _pickle.PicklingError: Can't pickle <class 'MyProxy'>: import of module <property object at 0x7f42833a50b0> failed

The factory in this case is a callable, pickleable object that returns a large numpy array, but I do not want to include the large array in the pickling (as done with the default behavior).

Is there a workaround for pickling the slots version of a Proxy? I am not using the lazy_object_proxy.cext.Proxy because I have run into some strange behavior with common numpy functions.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ionelmc/python-lazy-object-proxy/issues/57, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA7TXN3Y7NQW5XFKLAUTYDTK4H4ZANCNFSM43VURZ2A .

[gpauloski]

import pickle as pkl
from lazy_object_proxy.slots import Proxy

class Factory():
    def __init__(self, obj):
        self.obj = obj

    def __call__(self):
        return self.obj

class MyProxy(Proxy):
    def __reduce__(self):
        return MyProxy, (self.__factory__,)

    def __reduce_ex__(self, protocol):
        return MyProxy, (self.__factory__,)

p = MyProxy(Factory([1, 2, 3]))

p_pkl = pkl.dumps(p)  # Raises error
p = pkl.loads(p_pkl)

I am using lazy-object-proxy 1.6.0 and Python 3.7.10.

[ionelmc]

Well yes it breaks cause slots.Proxy defines a property for __module__ and that confuses pickle a lot.

There are two ways to solve this:

• use simple.Proxy
• use a trampoline like so:

def proxy_trampoline(factory):
    return MyProxy(factory)

class MyProxy(Proxy):
    def __reduce__(self):
        return proxy_trampoline, (object.__getattribute__(self, '__factory__'), )
    def __reduce_ex__(self, protocol):
        return proxy_trampoline, (object.__getattribute__(self, '__factory__'), )

[gpauloski]

That make sense, and the trampoline works well. Thanks for the help.