search

pangeo-data / rechunker

Disk-to-disk chunk transformation for chunked arrays.
https://rechunker.readthedocs.io/
MIT License
162 stars 25 forks source link

ZeroDivisionError in L70 of api.py #126

Open norlandrhagen opened 1 year ago

norlandrhagen commented 1 year ago

Hi there,

I'm having an issue when trying to rechunk a 4d zarr store. In the traceback at L70 of algorithm.py (70 headroom = max_mem // chunk_mem), I'm getting the error: ZeroDivisionError: integer division or modulo by zero.

The snippet below should be a MRE.


import xarray as xr
import fsspec
import rechunker 
import zarr 

store_url = "gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Omon/thetao/gn/v20190429"
ds = xr.open_dataset(store_url, engine='zarr', chunks={}, decode_cf=False)
group = zarr.open_consolidated(store_url)

chunks_dict = {
'i': {'i': 128},
 'j': {'j': 128},
 'latitude': {'i': 128, 'j': 128},
 'lev': {'lev': -1},
 'lev_bnds': {'lev': -1, 'bnds': -1},
 'longitude': {'i': 128, 'j': 128},
 'thetao': {'i': 128, 'j': 128, 'time': 2, 'lev': -1},
 'time': {'time': 2},
 'time_bnds': {'time': 2, 'bnds': -1},
 'vertices_latitude': {'i': 128, 'j': 128, 'vertices': -1},
 'vertices_longitude': {'i': 128, 'j': 128, 'vertices': -1}
}

tmp_mapper = fsspec.get_mapper('temp_store')
tgt_mapper = fsspec.get_mapper('staging_store')

array_plan = rechunker.rechunk(group, chunks_dict, "1000MB", tgt_mapper, temp_store=tmp_mapper)

A few notes: 

    chunk_mem = itemsize * prod(chunks)
    if chunk_mem > max_mem:
        raise ValueError(f"chunk_mem {chunk_mem} > max_mem {max_mem}")
    headroom = max_mem // chunk_mem

    new_chunks = list(chunks)
    # only consolidate over these axes
    axes = sorted(chunk_limit_per_axis.keys())[::-1]
    for n_axis in axes:
        c_new = min(
            chunks[n_axis] * headroom, shape[n_axis], chunk_limit_per_axis[n_axis]
        )
        # print(f'  axis {n_axis}, {chunks[n_axis]} -> {c_new}')
        new_chunks[n_axis] = c_new
        chunk_mem = itemsize * prod(new_chunks)
        headroom = max_mem // chunk_mem

Traceback:

```python --------------------------------------------------------------------------- ZeroDivisionError Traceback (most recent call last) Cell In[11], line 2 1 # %pdb on ----> 2 array_plan = rechunker.rechunk(group, chunks_dict, "1000MB", tgt_mapper, temp_store=tmp_mapper) File ~/opt/anaconda3/envs/install/envs/ncviewjs/lib/python3.10/site-packages/rechunker/api.py:302, in rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options, executor) 299 if isinstance(executor, str): 300 executor = _get_executor(executor) --> 302 copy_spec, intermediate, target = _setup_rechunk( 303 source=source, 304 target_chunks=target_chunks, 305 max_mem=max_mem, 306 target_store=target_store, 307 target_options=target_options, 308 temp_store=temp_store, 309 temp_options=temp_options, 310 ) 311 plan = executor.prepare_plan(copy_spec) 312 return Rechunked(executor, plan, source, intermediate, target) File ~/opt/anaconda3/envs/install/envs/ncviewjs/lib/python3.10/site-packages/rechunker/api.py:457, in _setup_rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options) 454 copy_specs = [] 456 for array_name, array_target_chunks in target_chunks.items(): --> 457 copy_spec = _setup_array_rechunk( ... ---> 70 headroom = max_mem // chunk_mem 72 if headroom == 1: 73 break ZeroDivisionError: integer division or modulo by zero ```

Results of conda list:

``` # Name Version Build Channel adal 1.2.7 pyhd8ed1ab_0 conda-forge adlfs 2022.11.2 pyhd8ed1ab_0 conda-forge aiohttp 3.8.3 py310h90acd4f_1 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aiosqlite 0.18.0 pypi_0 pypi alembic 1.9.1 pyhd8ed1ab_0 conda-forge anyio 3.6.2 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pypi_0 pypi appnope 0.1.3 pyhd8ed1ab_0 conda-forge apprise 1.2.1 pypi_0 pypi argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py310h90acd4f_3 conda-forge asciitree 0.3.3 py_2 conda-forge asgi-lifespan 2.0.0 pypi_0 pypi asttokens 2.2.1 pyhd8ed1ab_0 conda-forge async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge asyncpg 0.27.0 py310h90acd4f_1 conda-forge attrs 22.2.0 pyh71513ae_0 conda-forge awscli 1.27.43 py310h2ec42d9_0 conda-forge azure-core 1.26.1 pyhd8ed1ab_0 conda-forge azure-datalake-store 0.0.51 pyh9f0ad1d_0 conda-forge azure-identity 1.12.0 pyhd8ed1ab_0 conda-forge azure-storage-blob 12.14.1 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge bcrypt 4.0.1 pypi_0 pypi beautifulsoup4 4.11.1 pyha770c72_0 conda-forge bleach 5.0.1 pyhd8ed1ab_0 conda-forge blinker 1.5 pyhd8ed1ab_0 conda-forge bokeh 2.4.3 pyhd8ed1ab_3 conda-forge boto3 1.26.43 pyhd8ed1ab_0 conda-forge botocore 1.29.43 pyhd8ed1ab_0 conda-forge brotlipy 0.7.0 py310h90acd4f_1005 conda-forge bzip2 1.0.8 h0d85af4_4 conda-forge c-ares 1.18.1 h0d85af4_0 conda-forge ca-certificates 2022.12.7 h033912b_0 conda-forge cachetools 5.2.0 pyhd8ed1ab_0 conda-forge certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cf_xarray 0.7.6 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310ha78151a_3 conda-forge cfgv 3.3.1 pypi_0 pypi cftime 1.6.2 py310h936d966_1 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge click 8.1.3 unix_pyhd8ed1ab_2 conda-forge cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge comm 0.1.2 pyhd8ed1ab_0 conda-forge commonmark 0.9.1 pypi_0 pypi contourpy 1.0.6 pypi_0 pypi coolname 2.2.0 pypi_0 pypi croniter 1.3.8 pypi_0 pypi cryptography 39.0.0 py310hdd0c95c_0 conda-forge cycler 0.11.0 pypi_0 pypi cytoolz 0.12.0 py310h90acd4f_1 conda-forge dask 2022.12.1 pyhd8ed1ab_0 conda-forge dask-core 2022.12.1 pyhd8ed1ab_0 conda-forge dateparser 1.1.5 pypi_0 pypi debugpy 1.6.5 py310h7a76584_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge dictdiffer 0.9.0 pyhd8ed1ab_0 conda-forge distlib 0.3.6 pypi_0 pypi distributed 2022.12.1 pyhd8ed1ab_0 conda-forge docker 6.0.1 pypi_0 pypi docutils 0.15.2 py310h2ec42d9_6 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge fastapi 0.88.0 pyhd8ed1ab_0 conda-forge fasteners 0.17.3 pyhd8ed1ab_0 conda-forge fastprogress 1.0.3 pypi_0 pypi filelock 3.9.0 pypi_0 pypi flit-core 3.8.0 pyhd8ed1ab_0 conda-forge fonttools 4.38.0 pypi_0 pypi freetype 2.12.1 h3f81eb7_1 conda-forge frozenlist 1.3.3 py310h90acd4f_0 conda-forge fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge gcsfs 2022.11.0 pyhd8ed1ab_0 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge glib 2.74.1 hbc0c0cd_1 conda-forge glib-tools 2.74.1 hbc0c0cd_1 conda-forge google-api-core 2.11.0 pyhd8ed1ab_0 conda-forge google-auth 2.15.0 pyh1a96a4e_0 conda-forge google-auth-oauthlib 0.8.0 pyhd8ed1ab_0 conda-forge google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge google-cloud-storage 2.7.0 pyh1a96a4e_0 conda-forge google-crc32c 1.1.2 py310h6f7428f_4 conda-forge google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge googleapis-common-protos 1.57.1 pyhd8ed1ab_0 conda-forge greenlet 2.0.1 py310h7a76584_0 conda-forge griffe 0.25.3 pypi_0 pypi grpcio 1.51.1 py310h0f91fd3_0 conda-forge gst-plugins-base 1.21.3 h37e1711_1 conda-forge gstreamer 1.21.3 h1d18e73_1 conda-forge gunicorn 20.1.0 py310h2ec42d9_3 conda-forge h11 0.14.0 pyhd8ed1ab_0 conda-forge h2 4.1.0 pypi_0 pypi heapdict 1.0.1 py_0 conda-forge hpack 4.0.0 pypi_0 pypi httpcore 0.16.3 pypi_0 pypi httpx 0.23.3 pypi_0 pypi hyperframe 6.0.1 pypi_0 pypi icu 70.1 h96cf925_0 conda-forge identify 2.5.12 pypi_0 pypi idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.0.0 pyha770c72_0 conda-forge importlib_resources 5.10.2 pyhd8ed1ab_0 conda-forge intake 0.6.6 pypi_0 pypi intake-esm 2022.9.18 pypi_0 pypi ipykernel 6.19.4 pyh736e0ef_0 conda-forge ipython 8.8.0 pyhd1c38e8_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 8.0.4 pyhd8ed1ab_0 conda-forge isodate 0.6.1 pyhd8ed1ab_0 conda-forge jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jmespath 1.0.1 pyhd8ed1ab_0 conda-forge jpeg 9e hac89ed1_2 conda-forge jsonpatch 1.32 pypi_0 pypi jsonpointer 2.3 pypi_0 pypi jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge jupyter 1.0.0 py310h2ec42d9_8 conda-forge jupyter_client 7.4.8 pyhd8ed1ab_0 conda-forge jupyter_console 6.4.4 pyhd8ed1ab_0 conda-forge jupyter_core 5.1.2 py310h2ec42d9_0 conda-forge jupyter_events 0.4.0 pyhd8ed1ab_0 conda-forge jupyter_server 2.0.6 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.3 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 3.0.5 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.4 pypi_0 pypi krb5 1.20.1 h049b76e_0 conda-forge kubernetes 25.3.0 pypi_0 pypi lcms2 2.14 h29502cd_1 conda-forge lerc 4.0.0 hb486fe8_0 conda-forge libabseil 20220623.0 cxx17_h844d122_6 conda-forge libblas 3.9.0 16_osx64_openblas conda-forge libcblas 3.9.0 16_osx64_openblas conda-forge libclang 13.0.1 default_he082bbe_0 conda-forge libcrc32c 1.1.2 he49afe7_0 conda-forge libcxx 14.0.6 hccf4f1f_0 conda-forge libdeflate 1.14 hb7f2c08_0 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libgfortran 5.0.0 11_3_0_h97931a8_27 conda-forge libgfortran5 11.3.0 h082f757_27 conda-forge libglib 2.74.1 h4c723e1_1 conda-forge libgrpc 1.51.1 h966d1d5_0 conda-forge libiconv 1.17 hac89ed1_0 conda-forge libjpeg-turbo 2.1.4 hb7f2c08_0 conda-forge liblapack 3.9.0 16_osx64_openblas conda-forge libllvm13 13.0.1 h64f94b2_2 conda-forge libogg 1.3.4 h35c211d_1 conda-forge libopenblas 0.3.21 openmp_h429af6e_3 conda-forge libopus 1.3.1 hc929b4f_1 conda-forge libpng 1.6.39 ha978bb4_0 conda-forge libpq 15.1 h3640bf0_2 conda-forge libprotobuf 3.21.12 hbc0c0cd_0 conda-forge libsodium 1.0.18 hbcb3906_1 conda-forge libsqlite 3.40.0 ha978bb4_0 conda-forge libtiff 4.5.0 h6268bbc_0 conda-forge libvorbis 1.3.7 h046ec9c_0 conda-forge libwebp-base 1.2.4 h775f41a_0 conda-forge libxcb 1.13 h0d85af4_1004 conda-forge libzlib 1.2.13 hfd90126_4 conda-forge llvm-openmp 15.0.6 h61d9ccf_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4 4.2.0 py310h8d4e1d9_0 conda-forge lz4-c 1.9.3 he49afe7_1 conda-forge mako 1.2.4 pyhd8ed1ab_0 conda-forge markdown 3.4.1 pypi_0 pypi markupsafe 2.1.1 py310h90acd4f_2 conda-forge matplotlib 3.6.2 pypi_0 pypi matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mistune 2.0.4 pyhd8ed1ab_0 conda-forge msal 1.20.0 pyhd8ed1ab_0 conda-forge msal_extensions 1.0.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.4 py310ha23aa8a_1 conda-forge msrest 0.7.1 pyhd8ed1ab_0 conda-forge multidict 6.0.4 py310h90acd4f_0 conda-forge mypy_extensions 0.4.3 py310h2ec42d9_6 conda-forge mysql-common 8.0.31 hc4b2c72_0 conda-forge mysql-libs 8.0.31 h8658499_0 conda-forge nbclassic 0.4.8 pyhd8ed1ab_0 conda-forge nbclient 0.7.2 pyhd8ed1ab_0 conda-forge nbconvert 7.2.7 pyhd8ed1ab_0 conda-forge nbconvert-core 7.2.7 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 7.2.7 pyhd8ed1ab_0 conda-forge nbformat 5.7.1 pyhd8ed1ab_0 conda-forge ncurses 6.3 h96cf925_1 conda-forge ncviewjs-backend 0.0.post28 pypi_0 pypi nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge netcdf4 1.6.2 pypi_0 pypi nodeenv 1.7.0 pypi_0 pypi notebook 6.5.2 pyha770c72_1 conda-forge notebook-shim 0.2.2 pyhd8ed1ab_0 conda-forge nspr 4.35 hea0b92c_0 conda-forge nss 3.78 ha8197d3_0 conda-forge numcodecs 0.11.0 py310h7a76584_1 conda-forge numpy 1.24.1 py310h1b7c290_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge openjpeg 2.5.0 h13ac156_2 conda-forge openssl 3.0.7 hfd90126_1 conda-forge orjson 3.8.5 pypi_0 pypi packaging 22.0 pyhd8ed1ab_0 conda-forge pandas 1.5.2 py310hecf8f37_0 conda-forge pandoc 2.19.2 h694c41f_1 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge paramiko 2.12.0 pypi_0 pypi parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.3.0 pyhd8ed1ab_0 conda-forge pathspec 0.10.3 pypi_0 pypi pcre2 10.40 h1c4e4bc_0 conda-forge pendulum 2.1.2 pypi_0 pypi pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.4.0 py310hab5364c_0 conda-forge pip 22.3.1 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 2.6.2 pyhd8ed1ab_0 conda-forge ply 3.11 py_1 conda-forge portalocker 2.6.0 py310h2ec42d9_1 conda-forge pre-commit 2.21.0 pypi_0 pypi prefect 2.7.7 pypi_0 pypi prometheus_client 0.15.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.36 pyha770c72_0 conda-forge prompt_toolkit 3.0.36 hd8ed1ab_0 conda-forge protobuf 4.21.12 py310h7a76584_0 conda-forge psutil 5.9.4 py310h90acd4f_0 conda-forge psycopg2 2.9.3 py310h31f19fa_2 conda-forge pthread-stubs 0.4 hc929b4f_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pydantic 1.10.4 py310h90acd4f_0 conda-forge pygments 2.14.0 pyhd8ed1ab_0 conda-forge pyjwt 2.6.0 pyhd8ed1ab_0 conda-forge pynacl 1.5.0 pypi_0 pypi pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pypi_0 pypi pyqt 5.15.7 py310hdd03f62_2 conda-forge pyqt5-sip 12.11.0 py310h415000c_2 conda-forge pyrsistent 0.19.3 py310h90acd4f_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.8 he7542f4_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.4 pyhd8ed1ab_0 conda-forge python-slugify 7.0.0 pypi_0 pypi python_abi 3.10 3_cp310 conda-forge pytz 2022.7 pyhd8ed1ab_0 conda-forge pytz-deprecation-shim 0.1.0.post0 pypi_0 pypi pytzdata 2020.1 pypi_0 pypi pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyyaml 5.4.1 py310h90acd4f_4 conda-forge pyzmq 24.0.1 py310hf615a82_1 conda-forge qt-main 5.15.6 haeff654_5 conda-forge qtconsole 5.4.0 pyhd8ed1ab_0 conda-forge qtconsole-base 5.4.0 pyha770c72_0 conda-forge qtpy 2.3.0 pyhd8ed1ab_0 conda-forge re2 2022.06.01 hb486fe8_1 conda-forge readchar 4.0.3 pypi_0 pypi readline 8.1.2 h3899abd_0 conda-forge rechunker 0.5.0 pyhd8ed1ab_0 conda-forge regex 2022.10.31 pypi_0 pypi requests 2.28.1 pyhd8ed1ab_1 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rfc3986 1.5.0 pypi_0 pypi rich 13.0.0 pypi_0 pypi rsa 4.7.2 pyh44b312d_0 conda-forge ruamel.yaml 0.17.21 py310h90acd4f_2 conda-forge ruamel.yaml.clib 0.2.7 py310h90acd4f_1 conda-forge s3fs 0.4.2 py_0 conda-forge s3transfer 0.6.0 pyhd8ed1ab_0 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 65.6.3 pyhd8ed1ab_0 conda-forge sip 6.7.5 py310h7a76584_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge skyplane 0.2.1 pypi_0 pypi sniffio 1.3.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge sqlalchemy 1.4.41 py310h90acd4f_0 conda-forge sqlalchemy2-stubs 0.0.2a31 pyhd8ed1ab_0 conda-forge sqlite 3.40.0 h9ae0607_0 conda-forge sqlmodel 0.0.8 pyhd8ed1ab_0 conda-forge sshtunnel 0.4.0 pypi_0 pypi stack_data 0.6.2 pyhd8ed1ab_0 conda-forge starlette 0.22.0 pyhd8ed1ab_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.17.1 pyhd1c38e8_0 conda-forge text-unidecode 1.3 pypi_0 pypi tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h5dbffcc_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.2 py310h90acd4f_1 conda-forge traitlets 5.8.0 pyhd8ed1ab_0 conda-forge typer 0.7.0 pypi_0 pypi typing-extensions 4.4.0 hd8ed1ab_0 conda-forge typing_extensions 4.4.0 pyha770c72_0 conda-forge tzdata 2022.7 pypi_0 pypi tzlocal 4.2 pypi_0 pypi universal_pathlib 0.0.21 pyh191b570_0 conda-forge urllib3 1.26.13 pyhd8ed1ab_0 conda-forge uvicorn 0.20.0 py310h2ec42d9_1 conda-forge virtualenv 20.17.1 pypi_0 pypi wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.4.2 pyhd8ed1ab_0 conda-forge wheel 0.38.4 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.5 pyhd8ed1ab_0 conda-forge xarray 2022.12.0 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.9 h35c211d_0 conda-forge xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge yarl 1.8.2 py310h90acd4f_0 conda-forge zarr 2.13.3 pyhd8ed1ab_0 conda-forge zeromq 4.3.4 he49afe7_1 conda-forge zict 2.2.0 pyhd8ed1ab_0 conda-forge zipp 3.11.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hfd90126_4 conda-forge zstd 1.5.2 hfa58983_4 conda-forge ```

Wondering if anyone has any thoughts on why this might be happening.

Thanks!

rabernat commented 1 year ago

Thanks for sharing. Can reproduce it even more simply with

target_chunks = (2, -1, 128, 128)
rechunker.rechunk(group['thetao'], target_chunks, "1000MB", tgt_mapper, temp_store=tmp_mapper)

I'll investigate.

rabernat commented 1 year ago

So it looks like we can fix it with

target_chunks = (2, 45, 128, 128)

My impression is that we don't support the syntax -1 to mean "the full size of the dimension." This could easily be fixed. Or we could raise an error if you pass a chunk shape as a negative number.

norlandrhagen commented 1 year ago

Thanks for the wildly quick response @rabernat!

Didn't realize the -1 means the full size on the dim wasn't supported. Either of those options sounds like an improvement to me!

rabernat commented 1 year ago

PR welcome! 😉

The check would go here: https://github.com/pangeo-data/rechunker/blob/46db8074df3d77cac6cefd1fed0931366d207a9e/rechunker/algorithm.py#L68

Just replace the special value of -1 with the corresponding shape (i.e. swap -1 for 45 like I did above), something like

new_chunks = [s if c == -1 else c for c, s in zip(chunks, shape)]
norlandrhagen commented 1 year ago

Sounds great! I'll open up a PR.