search

HDFGroup / h5pyd

h5py distributed - Python client library for HDF Rest API
Other
110 stars 39 forks source link

Error with 1D chunk sizes #151

Closed alan113696 closed 11 months ago

alan113696 commented 11 months ago

When running hsget using h5pyd 0.15.0, I see the following error message:

creating dataset /SCENE/TEMPORAL/Coverage, shape: (2,), type: float64
Traceback (most recent call last):
  File "/home/centos/miniconda3/envs/hsds/bin/hsget", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5pyd/_apps/hsget.py", line 127, in main
    load_file(fin, fout, **kwargs)
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5pyd/_apps/utillib.py", line 1582, in load_file
    fin.visititems(object_create_helper)
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5pyd/_hl/group.py", line 1083, in visititems
    retval = func(h5path, parent)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5pyd/_apps/utillib.py", line 1545, in object_create_helper
    create_dataset(obj, ctx)
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5pyd/_apps/utillib.py", line 1161, in create_dataset
    dset = fout.create_dataset(dobj.name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5py/_hl/group.py", line 183, in create_dataset
    dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 104, in make_new_dset
    dcpl = filters.fill_dcpl(
           ^^^^^^^^^^^^^^^^^^
  File "/home/centos/miniconda3/envs/hsds/lib/python3.11/site-packages/h5py/_hl/filters.py", line 257, in fill_dcpl
    plist.set_chunk(chunks)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5p.pyx", line 467, in h5py.h5p.PropDCID.set_chunk
ValueError: chunksize must be a tuple.

This only occurs on 1D chunk sizes. I poked around the code, and it looks like expandChunk is the culprit. I changed the following line https://github.com/HDFGroup/h5pyd/blob/master/h5pyd/_apps/utillib.py#L932 to read return tuple(chunk_shape) as a work-around.

jreadey commented 11 months ago

Thanks for reporting this. Yes, expandChunks needs to return a tuple, Actually, expandChunks should only be used with hsload, not hsget. I've checked in a fix with the tuple(chunk_shape) and a check to not use expandChunks when the target is an h5 file. New version 0.15.1 is out on PyPI.

alan113696 commented 11 months ago

0.15.1 works great! Thanks for quickly pushing out the fix.