search

silx-kit / hdf5plugin

Set of compression filters for h5py
http://www.silx.org/doc/hdf5plugin/latest/
Other
64 stars 25 forks source link

Missing documentation for SZ3 parameters #317

Open mandelkow opened 1 month ago

mandelkow commented 1 month ago

Could someone add minimal documentation to explain the input parameters for SZ3?

hdf5plugin.SZ3(absolute=None, relative=None, norm2=None, peak_signal_to_noise_ratio=None)

The link provided ( https://szcompressor.org/ ) points to a slew of papers hidden behind a paywall.

The parameters absolute and relative may be analogous to the documentation of SZ!?

This paper Zhao et al. 2021 also quotes an equation for PSNR: PSNR = 10*log10( (max(D)-min(D))^2 / mean((D-D')^2)) [dB] i.e. peak-to-peak signal power over mean decompression error power in dB.

But what does norm2 do? (In my experiments the parameter had no effect at all!?!)

t20100 commented 1 month ago

Hi,

SZ3 comes from: https://github.com/szcompressor/SZ3 We'll update the link in the documentation to use this one instead.

Regarding the documentation of the different arguments, we had trouble with testing some of them (see https://github.com/silx-kit/hdf5plugin/pull/241, https://github.com/silx-kit/hdf5plugin/pull/224#discussion_r1037102334) and so we decided to document only the one that we could test relieable, but it looks norm2 is missing in the doc. There is actually a warning for the others:

https://github.com/silx-kit/hdf5plugin/blob/86d6cfe44697dcecf34475a5550e256bb3c3ab1c/src/hdf5plugin/_filters.py#L680

@vasole do you recall what was/is the exact state of testing the different SZ3 modes?

For documentation of the different mode, see the SZ3 tool doc:

https://github.com/szcompressor/SZ3/blob/4bbe9df7e4bcb6ae6339fcb3033100da07fe7434/tools/sz3/sz3.cpp#L40-L54

About norm2, it is explained here: https://github.com/silx-kit/hdf5plugin/blob/86d6cfe44697dcecf34475a5550e256bb3c3ab1c/src/SZ3/tools/sz3/sz3.config#L34

t20100 commented 1 month ago

In #314, I replaced the https://szcompressor.org/ by links to the repository. For the documentation of the parameters, it will require a closer look, so not for now for me.

t20100 commented 1 month ago

Regarding norm2, this comment https://github.com/silx-kit/hdf5plugin/pull/290#discussion_r1464663381 points to the implementation of the conversion of norm2 to an absolute error bound here: https://github.com/szcompressor/SZ3/blob/b3dac0c65a9a31a566ea4b264dad48427702d6db/include/SZ3/utils/Statistic.hpp#L42-L44

Without more documentation/information from the upstream SZ3 library, I'm +1 to keep hdf5plugin.SZ3 documentation as it is.