search

levyx / heliumdb

heliumdb provides a persisted python objected store capable of scaling to many terabytes of data
https://blu-corner.io
MIT License
1 stars 0 forks source link

Find another serialization library #6

Open arashn opened 4 years ago

arashn commented 4 years ago

Pickle, in some cases, may not be deterministic. We need to use another serialization library that ensures deterministic serialization of objects, and ideally one that also improves (de)serialization performance.

arashn commented 4 years ago

If we limit the keys and values to simple objects (string, int, list, dict, etc.), we can use a few other serialization libraries, like MessagePack or pyarrow (or even JSON or ujson), without requiring schema.