Open jllanfranchi opened 7 years ago
Other alternatives for this:
.npy
files, we will need to create a directory and each key as a filename and the contents be the value, either a .npy
file or .json
or somesuch. This gets ugly fast trying to translate a dict into a dir with files.
.npz
for multiple arrays in one file, but this doesn't help for arbitrary Python objectspip
installable, which is nice. Can use feather
file format, or the native format, or Apache Parquet(?)
serializable_state
in many core objects which produces a dict of simple Python datatypes (plus numpy types), it seems Arrow might be able to handle this as-is: http://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization
Not sure if this is fix-able. Pickle seems like a bad way to store really large maps (e.g. HDF5 would make more sense). But it might be a bug...
Alternately, could we integrate with npy binary-file format somehow? https://docs.scipy.org/doc/numpy/neps/npy-format.html Do we need to abandon pickle altogether?