Can't pickle <class 'google.protobuf.pyext._message.CMessage'>

[Caliber2]

I am using pathos (installed all dependencies from this repo only). But I get following Error- File "C:\Program Files\Python38\lib\pickle.py", line 558, in save f(self, obj) # Call unbound method with explicit self File "C:\Program Files\Python38\lib\site-packages\dill_dill.py", line 2077, in save_type StockPickler.save_global(pickler, obj, name=obj_name) File "C:\Program Files\Python38\lib\pickle.py", line 1068, in save_global raise PicklingError( _pickle.PicklingError: Can't pickle <class 'google.protobuf.pyext._message.CMessage'>: it's not found as google.protobuf.pyext._message.CMessage

[mmckerns]

Looks like you have encountered a unpicklable object. Could you provide a minimal example that reproduces your traceback?

[leogama]

It's from Google's Colab. This class is renamed for some reason:

>>> from google.protobuf.pyext import _message
>>> vars(_message)
{(...)
 'Message': google.protobuf.pyext._message.CMessage,
 (...)}
>>> _message.Message.__module__, _message.Message.__name__
('google.protobuf.pyext._message', 'CMessage')

Is it safe to trap simple renames like this and try something as the following?

try:
    pickler.save_global(klass)
except PicklingError as error:
    if "it's not found as" not in error.args[0] or klass.__module__ not in sys.modules:
        raise
    candidates = [name for name, obj in sys.modules[klass.__module__].__dict__.items() if obj is klass]
    if not candidates:
        raise
    if len(candidates) != 1:
        raise PicklingError("Can't pickle %r: type %s has multiple names in %s" % (klass, klass.__name__, klass.__module__)) from error
    pickler.save_global(klass, name=candidates[0])

And could this strategy also be applied to functions, for example?

[mmckerns]

Right, I think that's risky. What if instead of the package-level importing the object from the module (i.e. from foo import bar), we have module.foo.bar and module.bar where bar is a different object?

[leogama]

Right, I think that's risky. What if instead of the package-level importing the object from the module (i.e. from foo import bar), we have module.foo.bar and module.bar where bar is a different object?

Yes, I understand that stdlib pickle doesn't do it for some reason, but I also know that they don't do a lot of things that dill does for many reasons (e.g. to not rely on implementation specific details).

I'm not sure if I understand the scenario you are describing. Do you mean a class with the same name imported from another module overwriting the one in "module"? That would raise a different error:

PicklingError("Can't pickle %r: it's not found as %s.%s" % (obj, module_name, name))

Note that it tested for identity in the list comprehension. And the class, if found, is from that module because it was found in sys.modules using the __module__ attribute from the class itself.