kermitt2 / biblio-glutton

A high performance bibliographic information service: https://biblio-glutton.readthedocs.io
125 stars 16 forks source link

Experiment with alternative compression #63

Open kermitt2 opened 3 years ago

kermitt2 commented 3 years ago

We use snappy right now for LMDB stored records. There are other compression methods that might be relevant to small objects to get higher compression ratio and faster decompression (in principle we are less interested in compression speed).

See for instance https://morotti.github.io/lzbench-web as a benchmark study. zstd (which has a training mode) and lz4 might be more interesting than snappy for our usage.