search

jturner314 / ndarray-npy

.npy and .npz file format support for ndarray
https://docs.rs/ndarray-npy
Apache License 2.0
56 stars 18 forks source link

Support for save as memmap #63

Closed mkrzywda closed 2 months ago

mkrzywda commented 3 years ago

Hi @jturner314,

In 0.8 version I can't find direct possibility to save as memmap. So could you tell me about possibility save files as Memory-mapped like here https://numpy.org/doc/stable/reference/generated/numpy.memmap.html?

Regards, Maciej

jturner314 commented 3 years ago

Does a combination of write_zeroed_npy and view_mut_npy meet your needs? See this example.

mkrzywda commented 3 years ago

I need way in Rust similar to Python3. I want avoid other rust files format. I want combine Rust and output from them as input in Python3 code. So for me very important is using Numpy format.

from numpy.lib.format import open_memmap
X = open_memmap(name, mode='w+', dtype=dtype, shape=(max_size,)) # new file

The question of what are these Rust memmap objects, can it be loaded from numpy later seamlessly??

jturner314 commented 3 years ago

Memory mapping is just a special way to access files (as opposed to normal read/write calls); the files are in the same .npy format regardless of whether or not memory mapping is used. If all you want to do is write an .npy file which can be opened by NumPy (using numpy.lib.format.open_memmap or numpy.load), then the simplest way is to use the write_npy convenience function. If you want to use memory mapping instead of write calls, then you can use the example I linked in my previous comment.