Python: Loading to and from pickle results in bspline going missing

[joshjchayes]

Running Python3.6:

I have an object, Foo, which contains multiple objects, Bar. Each Bar object contains a BSpline. When I pickle Foo, if I load Foo back in the same ipython session, the BSpline is there and evaluates properly.

If I open a new ipython session in a different terminal, and unpickle Foo, Foo unpickles without error. However, when I run Foo.Bar.Bspline.eval(), then I get the exception Exception: b'Invalid reference to BSpline: Maybe it has been deleted?'

If I run Foo.Bar.BSpline.get_coefficients() then I get a similar message: ~/external_libraries/splinter/build/splinter-python/python/splinter/bspline.py in get_coefficients(self) 54 :return List of the coefficients of the BSpline 55 """ ---> 56 num_coefficients = splinter._call(splinter._get_handle().splinter_bspline_get_num_coefficients, self._handle) 57 coefficients_raw = splinter._call(splinter._get_handle().splinter_bspline_get_coefficients, self._handle) 58

~/external_libraries/splinter/build/splinter-python/python/splinter/splinter.py in _call(function, *args) 233 # TODO: Sometimes the string is correct, sometimes not. Investigate. 234 errorMsg = get_py_string(_get_handle().splinter_get_error_string()) --> 235 raise Exception(errorMsg) 236 237 return res

Exception: b'Invalid reference to BSpline: Maybe it has been deleted?'

Is there a known issue with pickling and unpickling BSplines? Have you got any suggestions on how I can solve this?

[gablank]

Hello @joshjchayes ! Sorry for the late reply, we've been quite busy lately.

The problem is that the SPLINTER backend is written in C/C++, and the Python part is simply a tiny wrapper making it easy to call into the C/C++ backend. Almost all calls to methods on Python objects are "sent" to the backend for evaluation, and the result is returned. When you create a BSpline object, most of the state of the BSpline is actually in the C/C++ part of the code base, which Pickle cannot possibly know how to serialize/deserialize. Pickle only sees the variable we use for storing the pointer to the "backend object".

What I suggest you do is use the BSpline.save method for saving the BSpline, and then load it again by passing the filename to BSpline.__init__. If you absolutely need Pickle to do all the work for you, I think you would have to use the copyreg module and create reduction and construction functions for the BSpline class. If you end up doing this we would be very interested in getting the changes merged upstream!

Sorry again for the late reply, hope you can get this to work sufficiently well for your needs.

[gablank]

Hello again.

I thought this was interesting so I hacked together an example on how to use the copyreg module for registering a reducer and constructor for the BSpline class.

Can you try to use the code from this example and see if this solves your problem?

[joshjchayes]

Hello! Thank you for the reply and suggestions. I've managed to find an alternative solution which involves defining getstate and setstate for the object I want to pickle. By doing this, it is possible to save the path to the bspline as an attribute, and avoid having pickle attempt to save the BSpline itself. This has also helped when pickling a logger. Obviously though, this doesn't work for pickling a lone BSpline, but your save and load functions are great in that case.

Sample code here:

def __getstate__(self):
    '''
    Overriding __getstate__ so that log and bsplines are not pickled
    '''
    d = self.__dict__.copy()
    if 'log' in d:
        d['log'] = d['log'].name
    if 'bspline' in d:
        d['bspline'] = None
    return d

def __setstate__(self, d):
    '''
    Modified __setstate__ so that a new log will be created and bsplines will be
    loaded from file
    '''
    if 'log' in d:
        d['log'] = logging.getLogger(d['log'])
    if 'bspline' in d:
        d['bspline'] = splinter.BSpline(d['bspline_path'])

    self.__dict__.update(d)

[gablank]

Pickle support for the BSpline class has been added in 1113eb124d355d5f239f0f135a19a9b32e9f57d5.