grantjenks / python-runstats

Python module for computing statistics and regression in a single pass.
http://grantjenks.com/docs/runstats/
Other
96 stars 19 forks source link

Cython broke pickle #3

Closed grantjenks closed 7 years ago

grantjenks commented 7 years ago
In [6]: from runstats.fast import Statistics

In [7]: len(pickle.dumps(Statistics(), protocol=2))
Out[7]: 35

In [8]: pickle.dumps(Statistics(), protocol=2)
Out[8]: '\x80\x02crunstats.fast\nStatistics\nq\x00)\x81q\x01.'

In [9]: stats = Statistics()

In [10]: for val in range(10): stats.push(val)

In [11]: stats.mean()
Out[11]: 4.5

In [12]: pickle.dumps(stats)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-aaf271f92de1> in <module>()
----> 1 pickle.dumps(stats)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in dumps(obj, protocol)
   1378 def dumps(obj, protocol=None):
   1379     file = StringIO()
-> 1380     Pickler(file, protocol).dump(obj)
   1381     return file.getvalue()
   1382 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in dump(self, obj)
    222         if self.proto >= 2:
    223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
    225         self.write(STOP)
    226 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in save(self, obj)
    304             reduce = getattr(obj, "__reduce_ex__", None)
    305             if reduce:
--> 306                 rv = reduce(self.proto)
    307             else:
    308                 reduce = getattr(obj, "__reduce__", None)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle Statistics objects

In [13]: pickle.dumps(stats, protocol=2)
Out[13]: '\x80\x02crunstats.fast\nStatistics\nq\x00)\x81q\x01.'

In [14]: pickle.loads(pickle.dumps(stats, protocol=2))
Out[14]: <runstats.fast.Statistics at 0x110c53e40>
grantjenks commented 7 years ago

See Issue #2 also. This also motivates get_params, set_params methods.

FlorianWilhelm commented 7 years ago

I also think that these methods will render pickle useless since they accomplish the same but more efficient. If pickle support is really necessary following docs might help https://docs.python.org/3/library/pickle.html#persistence-of-external-objects

grantjenks commented 7 years ago

I think we could map __getstate__ to get_state and __setstate__ to set_state and pickle would work again. I'm not sure if __reduce__ is also necessary.

FlorianWilhelm commented 7 years ago

I'll look into that tomorrow.

grantjenks commented 7 years ago

Deployed at 1.2.1