mbr / simplekv

A simple key-value store for binary data.
http://simplekv.readthedocs.io
MIT License
152 stars 50 forks source link

Add pickle protocol semantics to the simplekv interface #107

Open fjetter opened 4 years ago

fjetter commented 4 years ago

Some of the simplekv stores implement the pickle protocol and allow the safe serialization of the store (e.g. AzureBlockBlobStore see here. In this case, the pickle roundtrip restores the store both safely and semantically correct.

Some stores will simply fail when trying to be serialized, for instance (taken from the test suite, I guess there are more examples)

import pickle
from simplekv.cache import CacheDecorator
from simplekv.memory import DictStore
store = CacheDecorator(DictStore(), DictStore())
try:
    pickle.loads(pickle.dumps(store))
except RuntimeError: # max recursion depth exceeded
    pass

will cause a recursion depth runtime error.

And for yet other stores, they can theoretically be safely serialized but it isn't sensible to do so, for instance the memory store will not point to the same dictionary anymore

from simplekv.memory import DictStore
import pickle

store = DictStore()
store_roundtrip = pickle.loads(pickle.dumps(store))

store.put('key', b'key')
assert 'key' in store
assert not 'key' in store_roundtrip

For a user it is not immediately possible to identify in which category a given store belongs.

It would be a great addition to the simplekv interface to expose whether or not the store can be safely and sensibly serialised or not.

I would therefore propose an extension to the simplekv interface such that simplekv stores shall implement the pickle protocol from now on. The base class KeyValueStore would, by default, raise a NotImplementError which gives a generic exception with a short explanation about the culprits of serialising the stores.

On top of this, there is either a mixin or an attribute available which indicates that the given store can be safely serialized to allow the users to adapt their code depending on whether this is supported or not. This may be omitted if the stores themselves raise an appropriate NotImplementedError since users may check

pickleable = False
try:
    pickle.dumps(store)
    pickleable = True
except NotImplementError:
    pass