search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.57k stars 1.74k forks source link

Suggestion: update the embedded lz4 copy #8260

Open RJVB opened 5 years ago

RJVB commented 5 years ago

This is a (requested!) spin-off of #7560.

The LZ4 copy embedded in ZFS is about 5 years old by now, and LZ4 development has continued during that time. According to LZ4 dev(s), the current version has definitely seen improvements and at the very least, decompression speed should be faster (https://github.com/lz4/lz4/issues/625). There is now also the option to use persistent state (or dictionary) which should improve compression of small chunks, and "HC" compression. This sounds like several good reasons to upgrade the copy all the more since older versions will continue to work with datasets written by newer versions.

I'd love to start tinkering with this myself but there are too many ZFS-specific changes to the LZ4 code (and I'm actually trying to find more time to spend away from my keyboards). I can give a hand though, and lend a few neurones.

I also understand there are potential issues with set-ups using an L2ARC which make it less evident to update LZ4 before they have been addressed.

But maybe one or both of the following are possible nevertheless:

rincebrain commented 5 years ago

Just wanted to reference #3908 #5927 for prior discussions of related things, as well as #8044 for the L2ARC complications.

rlaager commented 4 years ago

I'm not sure who is working on this. Maybe @ahrens can comment. IIRC, this was discussed at an OpenZFS meeting a few months ago. Changing the decompressor is fine. But, there was a concern that changing the compressor would negatively impact dedup, as new duplicates of the same uncompressed data would now result in slightly different compressed data.