Open JanisGailis opened 6 years ago
Assigning @forman to figure out how to approach this and to make sure it doesn't disappear in noise.
We came across this issue recently when trying to open CCI Land Cover with dim size (lon=120000, lat=60000) prepared for the Copernicus Climate Data Store. Their convention is 0 <= lon < 360! OMG.
@JanisGailis happy to discuss a solution with you early next week.
Sure, I'm quite sure what the problem is.
When opening larger datasets that are defined on 0..360
longitude
grid, such asesa_msla_ext
given by Prosper, aMemoryError
is encountered on some machines (mine, 8GB).This is due to these lines:
https://github.com/CCI-Tools/cate/blob/a1e31c4673399a99d58786d1208c2ba13de138f2/cate/core/opimpl.py#L185-L193
Calling
var.values
tries to load the entire variable in memory as annp.ndarray
. The more memory safe way to approach this would be using a recursivegroupby
of the variable until onlylat,lon
or even justlon
remains and then do the value swapping. In a similar way how it is done incoregistration
:https://github.com/CCI-Tools/cate/blob/a1e31c4673399a99d58786d1208c2ba13de138f2/cate/ops/coregistration.py#L258
EDIT: Even when 'groupbying' down to
lat/lon
, don't call.values
on the data variable, as this will result in the new dataset being slowly converted to an xarray dataset consisting of many in-memorynumpy
datasets. Instead, do the conversion using some tricky indexing magic.