search

gallantlab / cottoncandy

sugar for s3
http://gallantlab.github.io/cottoncandy/
BSD 2-Clause "Simplified" License
33 stars 16 forks source link

Implement fast NPY upload #48

Open spopham opened 6 years ago

spopham commented 6 years ago

Rewrite upload_npy_array and download_npy_array to do in-memory and have a npz compressed option

Reference: https://github.com/numpy/numpy/blob/master/numpy/lib/format.py

anwarnunez commented 6 years ago

👍

anwarnunez commented 4 years ago

Just had a quick look b/c it was on my mind. This should be pretty straightforward (see below). This should really only be allowed for uncompressed arrays and GZIP compressed arrays.

https://github.com/numpy/numpy/blob/master/numpy/lib/format.py#L326 https://github.com/numpy/numpy/blob/master/numpy/lib/format.py#L430

Using these those two functions, I think one just needs to dump that at the beginning of the StringIO (https://github.com/gallantlab/cottoncandy/blob/df64302ccbc7c00571a165f8ad41a5af1bde6a47/cottoncandy/interfaces.py#L631 and https://github.com/gallantlab/cottoncandy/blob/df64302ccbc7c00571a165f8ad41a5af1bde6a47/cottoncandy/interfaces.py#L624)