search

cbm-fles / flesnet

CBM FLES Timeslice Building
7 stars 22 forks source link

Add zstd compression to archive files #132

Closed cuveland closed 10 months ago

cuveland commented 10 months ago

This PR modifies InputArchive, OutputArchive, and OutputArchiveSequence to support optional zstd compression of the data contents.

Whether data in an archive file is compressed is documented using a new member variable of the ArchiveDescriptor. Compression is switched off by default and can be enabled in tsclient using the "?c=zstd" parameter in the output uri. Reading from an archive file automatically decompresses if neccessary.

The zstd compression has a throughput of approximately 92 MB/s on the mFLES login node (using a single core).

oTTer-Chief commented 10 months ago

Looks all good to me.

Tested new files with old versions. They will give an FATAL: input stream error which is fine