search

ajdawson / eofs

EOF analysis in Python
http://ajdawson.github.io/eofs/
GNU General Public License v3.0
199 stars 60 forks source link

SVD error when using eof solver on dask array #122

Open emmomp opened 3 years ago

emmomp commented 3 years ago

I have been getting an SVD error when trying to call the xarray eof solver on a daskarray.

The error is due to the following line in eofs.standard: nonMissingIndex = np.where(np.logical_not(np.isnan(self._data[0])))[0]

np.where always fails and gives nans for dask arrays (see e.g. https://stackoverflow.com/questions/59957541/what-is-the-dask-equivalent-of-numpy-where)

Possibly related to #115 (although I can no longer see those notebooks).

Solved by calling .load() before calling the solver, but loses the advantages of dask.

lfery commented 2 years ago

I had the same problem and I was able to solve it by replacing this line in standard.py nonMissingIndex = np.where(np.logical_not(np.isnan(self._data[0])))[0] by

nonMissingIndex = dask.array.where(np.logical_not(np.isnan(self._data[0])))[0]
nonMissingIndex.compute_chunk_sizes()

I think it doesn't lose the advantages of dask as I have a dataset that doesn't fit in memory but the code worked even so. I hope that can be useful to anyone.