reproject loads whole cube into memory

radio-astro-tools / spectral-cube

Library for reading and analyzing astrophysical spectral data cubes

http://spectral-cube.rtfd.org

BSD 3-Clause "New" or "Revised" License

95 stars 61 forks source link

reproject loads whole cube into memory #827

Open keflavich opened 2 years ago

keflavich commented 2 years ago

This was noted in #506 and is noted in the docs, but we closed #506 without fixing this.

We should have the ability to reproject plane-by-plane without loading the whole cube into memory.

It's not obvious to me that dask can help with this at all - @astrofrog any ideas? Otherwise, we'll use the memmap machinery built for other reasons.

I'm going to need this soon so I'll prioritize it.

astrofrog commented 2 years ago

I think dask should be able to do this in the same way that we do e.g. convolution plane-by-plane? Note also that this is relevant: https://github.com/astropy/reproject/pull/214 - but I haven't had any time to look into it.

keflavich commented 1 year ago

https://github.com/radio-astro-tools/spectral-cube/pull/845 is my starting point