From a few tests I did, it seems that we can gain significant advantages in using compression schemes against HDF5 files.
The HDF5 file format implements in-file compression, but the supported schemes are only a few (gzip being the most interesting one). This PR lets the DataFile class read HDF5 files compressed using any of the following programs:
gzip
bzip2
zstd
xz
I did several tests considering the following factors:
Type of HDF5 file: were all the polarimeters turned on, or only a few?
Compression speed
Decompression speed
Compression ratio, i.e., how many times was the compressed file smaller than the original one? (Compression ratio 2: the file halved its size).
From the few tests I did, the best compressor is xz, which however takes ~20 minutes to compress a file (decompressing it is a matter of a few seconds). The best compromise is zstd, which produces files a bit larger than xz, but both compression and decompression are very fast (usually 10÷30 seconds for compression, ~4÷5 seconds for decompression). I would not bother with gzip, which produces sub-optimal compression ratios (.gz files are up to ×10 times larger than .zst and .xz files), and bzip2, which produces files only marginally smaller than gzip at the expense of a significantly larger processing time (several minutes).
Assuming we adopt zstd as the standard compressor, we can expect these results:
Compression ratio is ~2 if all the polarimeters are on, but this improves up to ~100 (yes, one hundred!) if only some polarimeters are being exercised. For instance, test files acquired while running phase switch I/V curves went from 4 GB down to ~30 MB.
The time required to access data in a file acquired when all the polarimeters are on will be larger, but on files like the phase switch I/V curves mentioned above, the processing speed is increased by a factor ~3, because zstd decompression is much faster than file I/O. (In other words, it takes less time to read 30 MB from disk and decompress them than to read 4 GB from disk.) Thus, it will be wise to choose which files should be compressed and which ones should be left untouched.
With this PR in place, you can compress HDF5 files from the command line with commands like:
$ zstd my_file.h5 # This creates file "my_file.h5.zst"
and then read them using
from striptease import DataFile
with DataFile("my_file.h5.zst") as inpf:
# Use inpf as a normal DataFile
…
The PR fixes hdf5db.py too, so that databases of HDF5 files include compressed files as well.
From a few tests I did, it seems that we can gain significant advantages in using compression schemes against HDF5 files.
The HDF5 file format implements in-file compression, but the supported schemes are only a few (
gzipbeing the most interesting one). This PR lets theDataFileclass read HDF5 files compressed using any of the following programs:gzipbzip2zstdxzI did several tests considering the following factors:
From the few tests I did, the best compressor is
xz, which however takes ~20 minutes to compress a file (decompressing it is a matter of a few seconds). The best compromise iszstd, which produces files a bit larger thanxz, but both compression and decompression are very fast (usually 10÷30 seconds for compression, ~4÷5 seconds for decompression). I would not bother withgzip, which produces sub-optimal compression ratios (.gzfiles are up to ×10 times larger than.zstand.xzfiles), andbzip2, which produces files only marginally smaller thangzipat the expense of a significantly larger processing time (several minutes).Assuming we adopt
zstdas the standard compressor, we can expect these results:zstddecompression is much faster than file I/O. (In other words, it takes less time to read 30 MB from disk and decompress them than to read 4 GB from disk.) Thus, it will be wise to choose which files should be compressed and which ones should be left untouched.With this PR in place, you can compress HDF5 files from the command line with commands like:
and then read them using
The PR fixes
hdf5db.pytoo, so that databases of HDF5 files include compressed files as well.