I've changed the behaviour of the workflow. I found astropy to be very memory hungry, so we now roll our own version of convolution via fourier.
For the sample set of 16 acquisitions (mixture of Landat's 5, 7 & 8, and Sentinel-2 A & B), all running in parallel on a single node was using ~256GB of memory. Now it uses ~56GB.
I've changed the behaviour of the workflow. I found astropy to be very memory hungry, so we now roll our own version of convolution via fourier.
For the sample set of 16 acquisitions (mixture of Landat's 5, 7 & 8, and Sentinel-2 A & B), all running in parallel on a single node was using ~256GB of memory. Now it uses ~56GB.
I've also added pyfftw to the dependencies list.