NCAS-CMS / cf-python

A CF-compliant Earth Science data analysis library
http://ncas-cms.github.io/cf-python
MIT License
120 stars 19 forks source link

Chunks error raised whilst regridding #665

Closed davidhassell closed 1 year ago

davidhassell commented 1 year ago

Foe version 3.14.0 to 3.15.1, regrid operations with Field.regrids or Field.regridc whose regrid axes are chunked with more than one chunk will fail as follows:

>>> obs.data.chunks   # axes 1 and 2 are the regrid  axes
((322, 322, 322, 322, 164), (322, 38), (322, 322, 76)) 
>>> obs_regrid = obs.regrids(model, method="linear")
File ~/miniconda3/lib/python3.10/site-packages/dask/array/rechunk.py:307, in rechunk(x, chunks, threshold, block_size_limit, balance, method)
    305 if isinstance(chunks, (tuple, list)):
    306     chunks = tuple(lc if lc is not None else rc for lc, rc in zip(chunks, x.chunks))
--> 307 chunks = normalize_chunks(
    308     chunks, x.shape, limit=block_size_limit, dtype=x.dtype, previous_chunks=x.chunks
    309 )
    311 # Now chunks are tuple of tuples
    312 ndim = x.ndim

File ~/miniconda3/lib/python3.10/site-packages/dask/array/core.py:3128, in normalize_chunks(chunks, shape, limit, dtype, previous_chunks)
   3120         raise ValueError(
   3121             "Input array has %d dimensions but the supplied "
   3122             "chunks has only %d dimensions" % (len(shape), len(chunks))
   3123         )
   3124     if not all(
   3125         c == s or (math.isnan(c) or math.isnan(s))
   3126         for c, s in zip(map(sum, chunks), shape)
   3127     ):
-> 3128         raise ValueError(
   3129             "Chunks do not add up to shape. "
   3130             "Got chunks=%s, shape=%s" % (chunks, shape)
   3131         )
   3133 return tuple(
   3134     tuple(int(x) if not math.isnan(x) else np.nan for x in c) for c in chunks
   3135 )

ValueError: Chunks do not add up to shape. Got chunks=((322, 322, 322, 322, 164), (-1,), (-1,)), shape=(1452, 360, 720)

The problem lies in cf.Data._regrid rechunking the regrid axes to, (-1,) as opposed to the correct -1.

PR to follow.

bewithankit commented 1 year ago

Thanks David!