Open aulemahal opened 3 years ago
Thanks for the bug report.
So I'm guessing rechunker is trying to put everything in a single chunk?
This should definitely not happen unless your total dataset size is < max_mem
, which is not the case here.
I tried to reproduce your issue but could not
import zarr
from dask.diagnostics import ProgressBar
from rechunker import rechunk
shape = (176, 226, 55115)
source = zarr.ones(shape, chunks=(20, 20, 55115), dtype='f8', store='tmp-data/source.zarr', overwrite=True)
rechunked = rechunk(source, (80, 60, 365), '3GB', 'tmp-data/target.zarr',
target_options=dict(overwrite=True))
assert rechunked._intermediate is None # no intermediate
with ProgressBar():
rechunked.execute()
Could you share a bit more detail about your input data and the exact code your are using to call rechunker?
Hi! Thanks for the very usefull package! I thinks I found a bug in the chunks choice mechanism:
My input dataset has shape (176, 226, 55115) with chunks (20, 20, 55115). The requested output chunks are (80, 60, 365). I allowed 3GB of
max_mem
, and there is a temp store.Rechunking fails with : (elided traceback)
Turns out 55115 176 226 = 2192254240 = 8 Go (float32) and is slightly over the number in the error message (by 2%). So I'm guessing rechunker is trying to put everything in a single chunk? Even though this is way above
max_mem
. Also, I never asked forBlosc
encoding, so I guess it is automatic? Not a problem, but it seems a smaller chunk should be chosen in that case.