search

Applied-GeoSolutions / multitemporal

Efficient, chainable time series processing of raster stacks.
GNU General Public License v2.0
4 stars 3 forks source link

enable chunking of images #42

Open ircwaves opened 5 years ago

ircwaves commented 5 years ago

Description:

Given a temporally dense time series, over a wide enough area, multitemporal's memory requirements can move into the realm of tooo much. Especially in the case where you will output an entire time series (after working some magic on it) -- that sends things passed 20GiB.

Current implementation

reads in block-wise, but holds all the data until the end and writes out everything in one shot.

Proposed resolution:

write out data block-wise as well.

@bhbraswell @justinfisk -- seems reasonable, no?

bhbraswell commented 5 years ago

@bhbraswell @justinfisk -- seems reasonable, no?

That seems totally reasonable and good as far as can tell.

ircwaves commented 5 years ago

offline, @justinfisk mentioned that there might have been some reason that the output was not done in chunks. Is that ringing any bells for either of you?

bhbraswell commented 5 years ago

My recollection is that after some initial discussions, at which point the current framework already existed, I just didn't want to write the code that would save and then reassemble the chunks. So, basically laziness. As long as each step in the processing chain has the data it needs then I can't think of a reason not to keep everything as chunks until the end if that is more performant.