search

getkeops / keops

KErnel OPerationS, on CPUs and GPUs, with autodiff and without memory overflows
https://www.kernel-operations.io
MIT License
1.03k stars 65 forks source link

[Request] Numpy memmap support #276

Closed L-Reichardt closed 1 year ago

L-Reichardt commented 1 year ago

Hello,

Your library has been amazing working with large pointclouds. Currently I am working with 4d-pointclouds (temporally assembled from single scans) and some sequences easily exceed 1.Bil points. One relevant field for this is 3D mapping.

Since these pointclouds are too large for RAM, GPU Memory, they are assembled instead on disk as numpy memmap arrays. Normally, PyKeOps would be the perfect tool to work with such clouds, however this numpy ndarray subclass is not supported. Is there a plan to support numpy memmaps?

Best Regards, L

L-Reichardt commented 1 year ago

For reference, so far I am reading the memmap array into multiple lazy tensors, step by step. Then I am concatenating all lazy tensors to achieve the desired results.

jeanfeydy commented 1 year ago

Hi @L-Reichardt,

Apologies for the delayed answer - we were very busy with teaching in Oct.-Dec. 2022 but are now back to research ;-) This is an interesting point: we will have a look, see how much work would be needed to support memmaps and update the thread when we have some more info.

Best regards, Jean

jeanfeydy commented 1 year ago

Hi @L-Reichardt,

Thinking about the issue, I think that we can merge it with #99 . I add the problem to our new shortlist of desirable features for 2023, and will update things in #99.

Thanks again for the link to memmap :-) Best regards, Jean

L-Reichardt commented 1 year ago

Hello @jeanfeydy,

thanks for the update on this issue.