search

pangeo-data / xESMF

Universal Regridder for Geospatial Data
http://xesmf.readthedocs.io/
MIT License
189 stars 34 forks source link

Issue encapsulating sps.COO weight mtx under xr.DataArray #193

Closed lemieuxbenedicte closed 8 months ago

lemieuxbenedicte commented 2 years ago

Quick description

 MemoryError: Unable to allocate 832. GiB for an array with shape
 (334167, 334000) and data type float64

This problem is encountered with the xesmf package while encapsulating the sparse sparse.COO matrix of the regridding weights under an xarray.DataArray structure. xarray fails interpreting the sparse.COO object hidden behind the numpy.ndarray type. It reserves Memory for the full size array instead. This can cause MemoryError failures and/or result in the use of an unecessary large amount of Memory.

Full description

Demonstration jupyter-notebook : raised error, package version, proposed fix, tests,...

huard commented 1 year ago

The link to the notebook doesn't work for me.

aulemahal commented 8 months ago

With xESMF 0.8.3, xarray 2023.12 and sparse 0.14, I haven't been able to reproduce the issue. My guess is that some incompability between xarray and sparse has now been fixed and the issue of weights being transformed to a numpy array unexpectedly is now gone.

I'll close this PR/issue for now. Hopefully, the fix is simply a question of upgrading the environment.