search

JuliaIO / HDF5.jl

Save and load data in the HDF5 file format from Julia
https://juliaio.github.io/HDF5.jl
MIT License
383 stars 139 forks source link

Handle partial chunks #1072

Closed 0 closed 1 year ago

0 commented 1 year ago

The extent is not guaranteed to be a multiple of chunk size. When partial chunks are present, the number of chunks should be rounded up in order to include them.

mkitti commented 1 year ago

Looks promising. I'm not sure why Windows seems to be a special case here.

Also run the formatter. See contrib/format.

0 commented 1 year ago

Reformatted.

I think the failure is caused by a newer version of libhdf5. The three Windows checks that fail all report "libhdf5 v1.12.2", but the others have lower versions: v1.10.4 and v1.12.1. (Confusingly, the ones that claim to be using v1.12.1 seem to get it through HDF5_jll v1.12.2+2, but that's probably a separate issue.) I can reproduce the failure locally by using the system version of libhdf5, which is newer than v1.12.1.

I may have found the relevant HDF5 commit, which was added between v1.12.1 and v1.12.2, and appears to change the logic for determining what constitutes a partial allocation. The failing test might be checking something that's too internal to libhdf5. Is it worth keeping it?

mkitti commented 1 year ago

The JLL versions do not necessary correspond to their upstream versions. See https://github.com/JuliaPackaging/Yggdrasil/blob/86f01f3d8b89c113347aec7e51f8105cd5293be1/H/HDF5/build_tarballs.jl#L9-L30 .

Could you modify the test so that it is dependent on the HDF5 C library version?

0 commented 1 year ago

Modified failing test to be version-dependent.

mkitti commented 1 year ago

Thanks