Open jshin47 opened 5 years ago
Hi @jshin47,
we had a PR in xtensor-io that implemented mmap'ing HDF5 files. Non-compressed numpy files would be just as straight forward. You can have a look here: https://github.com/QuantStack/xtensor-io/pull/18/files#diff-4f602a45ff1e0fd1ebc810d7566a0b98R175
I think this shows quite clearly how to do it.
Not sure if we want to add this in xtensor core, though ... but it could also live in xtensor-io.
Great, this is very helpful! I can do HDF5 instead of NPY for now. I do think having this kind of support would be really useful in scenarios involving, say, order book data where its quite a bit of data and you need to parallelize.
On Mon, Jan 21, 2019 at 1:22 PM Wolf Vollprecht notifications@github.com wrote:
Hi @jshin47 https://github.com/jshin47,
we had a PR in xtensor-io that implemented mmap'ing HDF5 files. Non-compressed numpy files would be just as straight forward. You can have a look here: https://github.com/QuantStack/xtensor-io/pull/18/files#diff-4f602a45ff1e0fd1ebc810d7566a0b98R175
I think this shows quite clearly how to do it.
Not sure if we want to add this in xtensor core, though ... but it could also live in xtensor-io.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QuantStack/xtensor/issues/1359#issuecomment-456162106, or mute the thread https://github.com/notifications/unsubscribe-auth/ABphwtluz1dBGS-K1ewx4KWVYyyeLJjkks5vFgV4gaJpZM4aJToe .
Since we support npy format in xtensor core, it would make sense to support the mmap'ed version here too.
probably we should have a easy way to do mmap_adapt
and then we could reuse that function in the npy loader.
It would be very useful for me if I could use a library like
mio
withxtensor
so I don't have to read in the entirenpy
file into memory. This would be useful in different parallelism scenarios.