search

serazing / xscale

Xscale a library of multi-dimensional signal processing tools using parallel computing
Apache License 2.0
25 stars 12 forks source link

Window.convolve consumes too much memory #3

Closed serazing closed 6 years ago

serazing commented 7 years ago

Those lines need to be improve in Window.convolve because it consumes too much memory

mask = self.obj.notnull()
if weights is None:
    weights = im.convolve(mask.astype(float), coeffs, mode=mode)
filled_data = self.obj.fillna(0.).data
serazing commented 6 years ago

The convolution with the mask is also mapped over the different chunks using map_overlap from dask.array. This do not create memory issue anymore.