mhe / pynrrd

Simple pure-python module for reading and writing nrrd files.
https://pynrrd.readthedocs.io/
MIT License
116 stars 51 forks source link

Support `np.memmap` #150

Open kwsp opened 2 months ago

kwsp commented 2 months ago

I am trying to migrate some large datasets to nrrd and indeed pynrrd, and one thing I desire is to have nrrd.read_data support np.memmap. All my data is saved raw without compression, and say I get gigabytes of data in a file, it's much faster for me to do a np.memmap where I'll then read the segment of data I need, rather than loading everything into memory first.

addisonElliott commented 2 months ago

Definitely sounds like a useful feature. Some questions/comments.

Sounds like it should only apply when encoding is raw. Should it apply to ASCII/text encodings? Not sure that makes sense.

Should the memmap be read-only? It's definitely odd to do nrrd.read() and then have a mutable object. I'm particularly concerned about if the size changes (is that possible with memmap?)

I'm leaning towards a separate function like read_memmap instead of adapting read.

I'm also unclear about Fortan vs C-style ordering, but hopefully the order parameter in np.memmap would suffice.

kwsp commented 2 months ago

@addisonElliott I agree it