search

graphite-project / graphite-web

A highly scalable real-time graphing system
http://graphite.readthedocs.org/
Apache License 2.0
5.89k stars 1.26k forks source link

[BUG] pickled Interval and IntervalSet objects returned from python 2 graphite web cannot be unpickled in python3 #2692

Closed drawks closed 3 years ago

drawks commented 3 years ago

Describe the bug The pickled data returned from the /metrics/find endpoint of graphite-web running on python2 cannot be unpickled by graphite-web running on python3.

To Reproduce While this can be observed by configuring an instance of graphite-web running on python 3 to point to any graphite-web hosts running on python2 as their "cluster servers". The minimal reproducible bug is easily arrived at in the python repl

$ python2.7
Python 2.7.16 (default, Aug  7 2019, 19:39:03)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-23)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import graphite.intervals
>>> i = graphite.intervals.Interval(1,10)
>>> s = graphite.intervals.IntervalSet([i])
>>> import pickle
>>> with open("py2.pickle", "wb") as f:
...     pickle.dump(s, f)
...
>>>
$ python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('py2.pickle', 'rb') as f:
...     contents=pickle.load(f)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
AttributeError: 'Interval' object has no attribute '__dict__'

Expected behavior I would expect that regardless of python version being run on graphite-web hosts that the wire protocols used to between them would be compatible.

Environment (please complete the following information):

Additional context The cause of this seems to be a difference in how slotted objects are stored in python2 and python3. If you modify the Interval and IntervalSet class definitions to not use slots the objects unpickle just fine on python3. A side effect of this problem is that graphite-web on python3 is incompatible with other graphite-web implementations (carbonapi for example) which use pickled objects that copy the official graphite-web.

While it seems unlikely that there are many mixed python2/3 graphite-web clusters in the wild there definitely are people using graphite-web as a front end to carbonapi that will be surprised to find that graphite-web running on python3 will not work for them.

drawks commented 3 years ago

After playing with this a little bit, I think the issue may be that __slots__ doesn't seem to actually do anything on python2 when using "old style" classes. Where python3 automatically creates "new style" classes regardless of the syntax. So on python2 both Interval and IntervalSethas a __dict__ attribute even though they have __slots__ defined. When you go to unpickle on python3 the shape of the object isn't the same any longer since the python3 version properly applies the __slots__ and expects an object with no __dict__ attribute.