FSST compression method

[AbstractiveNord]

Is your feature request related to a problem? Please describe. Main data type in ManticoreSearch is String. Compression techniques allows to increase hardware utilization, efficient use of memory and disk resources. It's good idea to check, can be FSST technique used for improving ManticoreSearch.

Describe the solution you'd like Test, measure and implement FSST string compression method.

Additional context Less data to read may provide better perfomance, so it's good to test FSST both to row-wise storage and columnar storage.

FSST Repository.

[tomatolog]

stored strings use docstore that already uses lz4 compression library and you already could use high level of the ocmpression for it as described at manual docstore_compression

[AbstractiveNord]

stored strings use docstore that already uses lz4 compression library and you already could use high level of the ocmpression for it as described at manual docstore_compression

Authors of FSST says that their method provides better compression ratio and better compression speed.

[sanikolaev]

Some notes after today's dev call:

• Docstore: may help, but provided we utilize lazy fetching, the effect may be invisible

• Columnar: can help with compression ratio and encoding (i.e. data write) performance:

  

  but can barely help with search performance.

[tomatolog]

Authors of FSST says that their method provides better compression ratio and better compression speed.

From their README.md I see the speed is the same

When compared to e.g. LZ4 (which is block-based), FSST further achieves similar decompression speed and compression speed, and better compression ratio.

The advantages that it could decompress the only strings without touching the whole block. And equality comparisons can be performed without decompressing.