Closed gvanhoy closed 1 year ago
Although some results suggest there should be faster ways to compress, in local tests, swapping between Pickle and numpy's tobytes() did not show a significant effect in the time it takes to generate/store Sig53 or WidebandSig53.
Numpy's tobytes() may be better to use if we want datasets to be more portable, but to enable portability, we also need to handle labels, which can be difficult to convert to binary in a way that's portable.
For portability, we can consider providing scripts for unpacking the data in other languages/environments.
Swap pickle for np.savez for raw data.