ENH: real-time compression?

[maxpietsch]

Dear Santa, It might be worth investing in a compression algorithm that offers a good compromise of compression ratio and performance for our data formats. I am thinking of something like LZ4 or zstd. C-Blosc (BloscLZ) might be a good candidate package: here is a multi-threaded example. All of the above are licensed under the BSD license.

Depending on disk and RAM IO speed and availability of CPU cycles, there should be little or no overhead or even a net speed up (benchmark of blosclz compared to memcpy()).

The speed difference of lz4 to gzip is dramatic, the compression ratio is pretty close. Test using the ubuntu gzip and lz4 compression on a 404MB .mif file:

time lz4 wm.mif
real    0m0.331s
user    0m0.224s
sys 0m0.104s

time gzip wm.mif
real    0m5.082s
user    0m4.924s
sys 0m0.152s

The file sizes are: 96M for wm.mif.lz4 and 91M for wm.mif.gz.

[jdtournier]

I reckon you've been a naughty boy, cos I don't think Santa's going to deliver on this one... 🎅

OK, to do this right, you'd need to:

• replicate core/file/gz.h to offer the equivalent lz4 wrapper
• replicate the core/image_io/gz.h and core/image_io/gz.cpp file to provide the image handlers to deal with these types of data
• replicate the core/formats/mrtrix_gz.cpp to handle mif.lz4. I assume there's no point in supporting nii.lz4 given no other package will be able to deal with it... (?)
• add tests to configure to check for the lz4 library, like it does for zlib - and presumably allow for failure, in which case support for this is disabled (like we do for libtiff). Unless this is a header-only library, in which case we could include it as-is like we do for JSON?

None of this is difficult, it just takes time...

[chrisadamsonmcri]

Bump. I was going to just post about this. It would be good to have zstd support, and use .nii.zst rather than .nii.gz generally. The compression and decompression is MUCH faster and the file sizes are smaller. Although I/O is a small part of an overall pipeline the savings add up over time.