search

silx-kit / hdf5plugin

Set of compression filters for h5py
http://www.silx.org/doc/hdf5plugin/latest/
Other
65 stars 25 forks source link

Updated embedded libraries: SZ3 #289

Open t20100 opened 8 months ago

t20100 commented 8 months ago

This PR aims at upgrading SZ3 to the latest release: https://github.com/szcompressor/SZ3/releases/tag/v3.1.8

t20100 commented 8 months ago

The failing test is the one reading an already exisiting file, so that was compressed with a previous version of the filter. The round-trip test pass. So the new version of the SZ3 filter cannot read file generated with previous versions of the filter....

vasole commented 8 months ago

So the new version of the SZ3 filter cannot read file generated with previous versions of the filter....

I guess we'll have to check if the SZ3 developers are aware of it.

t20100 commented 7 months ago

This Pr is pending for a new release of SZ3 (see https://github.com/szcompressor/SZ3/issues/50#issuecomment-1915052871)

t20100 commented 1 month ago

Updated to use SZ3 v3.2.0 which checks that the data was compressed with a compatible version of the library.

Unfortunately, the cd_values format has changed from a couple of arguments to the dump of an instance of a Config C++ class (conversion code).

This makes reading a HDF5 dataset compressed with a previous version of SZ3 abort the process because it cannot read the Config object. This defeats the check of the version by the library. For instance, test/test.py fails with:

testSZ3 (__main__.TestHDF5PluginRead)
  Test reading and witing SZ3 compressed data ... terminate called after throwing an instance of 'std::length_error'
    what():  vector::_M_default_append
  start H5Z_filter_sz3

Also, when writing a new HDF5 dataset, the current passed arguments must be changed, so as it is the hdf5plugin.test.TestHDF5PluginRW.testSZ3 test fails with:

start H5Z_sz3_set_local
terminate called after throwing an instance of 'std::length_error'
  what():  vector::_M_default_append
Aborted

It works when setting SZ3.filter_options = (). To make it work means re-implementing and the dump of the Config object to integers here in Python. Since this mechanism was not developed to be use outside of C++, I'm -1 to duplicate it here.

Finally, it would be good to check that the Config read/write works when writing with one endianess and reading with another.