Feature Request: Advanced compression features

[HittingSmoke]

I would like to see more fine-grained control of compression in the form of volume/target options and settable attributes.

Use case example for compression levels:

I have the following configuration:

2x NVMe drives in RAID 1 as a foreground target. 6x 5400 RPM disks for low cost/TB and energy efficiency as a background target with EC and 2 replicas enabled. "RAID 6".

The NVMe drives in the foreground target are configured with lz4 level 1 (fastest in all categories).

The spinning disks in the background target are configured with zstd level 19.

As can be seen in this benchmark from Peazip, zstd compression and decompression speeds do not scale at the same rate. Decompression speed remains largely unaffected by compression level. This is great for write-once read-often data. Or in this case, it's great for background compression where the impact of compression performance is not seen by the end user:

So when writing to this bcacefs array I would see incredible write performance. Then in the background at low CPU priority, the filesystem will write my data back to the slow spinning disks with extremely high compression. When reading from the spinning disks my read performance will have increased as the I/O requirements will be reduced, handed off to the CPU for decompression.

This is only the beginning of the potential possibilities for supporting extended compression settings. Take zstd dictionaries for example: https://facebook.github.io/zstd/#small-data

Imagine one enables a zstddict option. Then you could use zstd --train * on a directory. The outputted dictionary file could be stored in the folder as a hidden .zstddict file. Then when bcachefs is writing files to that directory a dramatic improvement in compression ratio could be achieved by using the dictionary if it's present.

zstd also has many other options like adaptive mode or long range mode that may be great for specific workloads which could be broken down by file or directory.

As compression tools get more complex it's not enough to just have different algorithms for different workloads. The compression tools themselves need to be able to be fine tuned for the workload.

[hyperfekt]

Under strong background compression it might be worthwhile to delay rebalancing until the device is on AC power. Not sure how this could be achieved.

[r4m0n]

Under strong background compression it might be worthwhile to delay rebalancing until the device is on AC power. Not sure how this could be achieved.

Sounds useful. Maybe a /sys flag to enable/disable background rebalancing? the rest could be handled with some simple scripts. Probably would need some limits on how much data can be left on the fg drive, if you write too much you'll need to push it to bg, even if you are on battery.