search

LLNL / Silo

Mesh and Field I/O Library and Scientific Database
https://silo.llnl.gov
Other
30 stars 24 forks source link

Fix HDF5 filter ids for hzip and fpzip filters built into Silo #427

Open markcmiller86 opened 1 day ago

markcmiller86 commented 1 day ago

Ugh...for some reason, the filter ids used by Silo library for two internal compression filters (hzip and fpzip) were set to use (H5Z_FILTER_RESERVED+1 and H5Z_FILTER_RESERVED+2).

This is bad on two fronts. Across HDF5 library versions, if H5Z_FILTER_RESERVED changes, the filter ids change. Those numbers must be constant across all time and versions. Now, I did check and it looks like H5Z_FILTER_RESERVED has had the value 256 in all versions of HDF5 from 1.14.4 going back to 1.8.0.

This means that hzip and fpzip filters, if they have ever been used (I honestly don't think they have been used) would have then been given ids 257 and 258 respectively. We should now set them to those values. We should check with hdf group and ask if that is problem for us to now ensure those numbers get officially allocated to hzip and fpzip, maybe branding these as "silo-hzip" and "silo-fpzip"

Alternatively, we assume those filters have never been used and select more appropriate ids for them now and install them into the Silo library. It looks like there is a bonified implementation of an fpzip filter, https://github.com/NCAR/fpzip_plugin

@qkoziol thoughts?

qkoziol commented 1 day ago

Agree that the reserved filter ID value hasn't changed from 256, as intended. The range of 0-255 are reserved for internal use by the HDF5 library, the range of 256-32767 are for public filters registered with the HDF Group, and the range of 32768-65535 are for filters that are used privately / internally within an organization (and shouldn't ever be used for HDF5 data intended to be shared publicly). So, those values could be safe from conflicting with others and after consulting the list of filters that have been registered with the HDF Group (here: https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md ), are currently still available for public filters.

I agree: contacting @derobins and registering them so that another filter isn't registered with those IDs is the best way to preserve the use of those ID values.