search

kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
505 stars 42 forks source link

use zstd compress dict for lz4 ? #9

Closed gcxfd closed 4 years ago

gcxfd commented 4 years ago

i saw the readme.md write

lmdb-store can optionally use off-thread LZ4 compression ..... Zstandard provides utilities for creating your own optimized shared dictionary

image

kriszyp commented 4 years ago

Is the issue here about getting more information about ZStandard (https://facebook.github.io/zstd/)?

gcxfd commented 4 years ago

i found the how to

https://github.com/lz4/lz4/releases/tag/v1.8.1.2

lz4 doesn't specify how to build a dictionary. All it says is that it can be any file up to 64 KB.
This approach is compatible with zstd dictionary builder, which can be instructed to create a 64 KB dictionary with this command :

zstd --train dirSamples/* -o dictName --maxdict=64KB