search

cloudpipe / cloudpickle

Extended pickling support for Python objects
Other
1.64k stars 167 forks source link

Is it possible to re-generate the source of pickled objects including encapsulated globals? #532

Open kesmit13 opened 5 months ago

kesmit13 commented 5 months ago

I know that I can pickle a dictionary or list of functions, then unpickle them and use the inspect.getsource to print the source of each of them, but that does not include the source of any globals that are referenced inside those functions. I'd like to be able to regenerate the source of all pickled objects to create runnable source. I have to think the globals are stored somewhere that can be inspected, but I wasn't able to find them.

makslevental commented 5 months ago

FWIW each function has an attribute __globals__ but you're going to end up pickling the whole world...

ogrisel commented 3 months ago

Do you have an illustrative yet minimal code snippet to explain what you need/want?

kesmit13 commented 3 months ago

I was hoping to find a way to take code like this:

import foo

def unused_func(z):
    return z ** z

def util_func(x):
    return x * 2

def pickled_func(y):
    return util_func(foo.square(y))

Then be able to pickle the pickled_func function, but also be able to reconstruct the source. In this case, that would generate the source for util_func and pickled_func, but leave out unused_func.

To be honest, this was mostly an attempt to get around the limitation that the Python version must be the exact version used when creating the pickled objects. If the source could be regenerated, it would at least work in the same minor Python version and possibly even more than that.

ogrisel commented 2 months ago

Nobody has worked on such a thing and it's quite different from the way cloudpickle has been built.

It might or might not be possible to achieve using Python inspection utils, but even if it is, would probably require a lot of work to implement and properly test and I have no plans to work on such a thing personally.