JuliaIO / HDF5.jl

Save and load data in the HDF5 file format from Julia
https://juliaio.github.io/HDF5.jl
MIT License
387 stars 140 forks source link

More informative errors when missing a filter #1014

Closed nhz2 closed 1 year ago

nhz2 commented 1 year ago

If the following is used to make a file with a compressed dataset.

using HDF5
using H5Zlz4
fn = "lz4_compressed.test_h5"
f = h5open(fn, "w")
data = zeros(100, 100)
ds = create_dataset(
         f, "lz4", datatype(data), dataspace(data); chunk=(100, 100), filters=Lz4Filter()
)
write(ds, data)
close(f)

Then in a fresh julia session, the file is read

using HDF5
fn = "lz4_compressed.test_h5"
f = h5open(fn)
read(f["lz4"])
close(f)

Before this PR you get the following error.

Unfortunately this error message doesn't say what filter is missing or how to install it.

ERROR: HDF5.API.H5Error: Error reading dataset /lz4
libhdf5 Stacktrace:
  [1] H5PL__find_plugin_in_path: Plugin for dynamically loaded library/Can't open directory or file
      can't open directory: /home/conda/feedstock_root/build_artifacts/hdf5_split_1638790369387/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/hdf5/plugin
   ⋮
Stacktrace:
 [1] macro expansion
   @ ~/.julia/packages/HDF5/T4H0V/src/api/error.jl:18 [inlined]
 [2] h5d_read(dataset_id::HDF5.Dataset, mem_type_id::HDF5.Datatype, mem_space_id::HDF5.Dataspace, file_space_id::HDF5.Dataspace, xfer_plist_id::HDF5.DatasetTransferProperties, buf::Matrix{Float64})
   @ HDF5.API ~/.julia/packages/HDF5/T4H0V/src/api/functions.jl:542
 [3] _generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Float64}, ::Nothing)
   @ HDF5 ~/.julia/packages/HDF5/T4H0V/src/readwrite.jl:168
 [4] generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Float64})
   @ HDF5 ~/.julia/packages/HDF5/T4H0V/src/readwrite.jl:134
 [5] read(obj::HDF5.Dataset)
   @ HDF5 ~/.julia/packages/HDF5/T4H0V/src/readwrite.jl:39
 [6] top-level scope
   @ REPL[4]:1

After this PR you get the following error.

ERROR: filter missing, filter id: 32004 name: HDF5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html
Try running `import H5Zlz4` to install this filter.

Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] ensure_filters_available(f::HDF5.Filters.FilterPipeline{HDF5.DatasetCreateProperties})
   @ HDF5.Filters ~/juliadev/HDF5/src/filters/filters.jl:459
 [3] _generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Float64}, ::Nothing)
   @ HDF5 ~/juliadev/HDF5/src/readwrite.jl:191
 [4] generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Float64})
   @ HDF5 ~/juliadev/HDF5/src/readwrite.jl:134
 [5] read(obj::HDF5.Dataset)
   @ HDF5 ~/juliadev/HDF5/src/readwrite.jl:39
 [6] top-level scope
   @ REPL[4]:1

caused by: HDF5.API.H5Error: Error reading dataset /lz4
libhdf5 Stacktrace:
  [1] H5PL__find_plugin_in_path: Plugin for dynamically loaded library/Can't open directory or file
      can't open directory: /home/conda/feedstock_root/build_artifacts/hdf5_split_1638790369387/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/hdf5/plugin
   ⋮
Stacktrace:
 [1] macro expansion
   @ ~/juliadev/HDF5/src/api/error.jl:18 [inlined]
 [2] h5d_read(dataset_id::HDF5.Dataset, mem_type_id::HDF5.Datatype, mem_space_id::HDF5.Dataspace, file_space_id::HDF5.Dataspace, xfer_plist_id::HDF5.DatasetTransferProperties, buf::Matrix{Float64})
   @ HDF5.API ~/juliadev/HDF5/src/api/functions.jl:542
 [3] _generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Float64}, ::Nothing)
   @ HDF5 ~/juliadev/HDF5/src/readwrite.jl:168
 [4] generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Float64})
   @ HDF5 ~/juliadev/HDF5/src/readwrite.jl:134
 [5] read(obj::HDF5.Dataset)
   @ HDF5 ~/juliadev/HDF5/src/readwrite.jl:39
 [6] top-level scope
   @ REPL[4]:1
nhz2 commented 1 year ago

I added a test for this, but it may not be such a good idea, because the test will fail if using H5Zlz4 runs before the test runs, even if the using H5Zlz4 happens in another testset.

mkitti commented 1 year ago

I recall that there is a way to unregister and reregister a filter plugin.

nhz2 commented 1 year ago

I recall that there is a way to unregister and reregister a filter plugin.

Thanks, I'll try rewriting the test using that.

mkitti commented 1 year ago

We have a HDF5.API.h5z_unregister.

I wonder if we should go even further. If we are in interactive mode, Base.isinteractive(), perhaps we should ask the user if they would like us to just install the filter package?

musm commented 1 year ago

Thank you!