search

taoensso / nippy

The fastest serialization library for Clojure
https://www.taoensso.com/nippy
Eclipse Public License 1.0
1.04k stars 60 forks source link

Question: does Nippy Zstandard support training? #168

Closed lowecg closed 4 months ago

lowecg commented 4 months ago

I see that Zstandard compression support has been added, which is great news.

We use Nippy to generate lots of small documents with similar data, which seems like a good fit for Zstandard training. Is training possible with Nippy?

ptaoussanis commented 4 months ago

@lowecg Hi there!

Is training possible with Nippy?

Not currently, at least. Nippy v3.4 uses aircompressor for its compression - and from a quick skim of the docs, I don't believe that it currently offers any form of explicit/opt-in training.

I'm not too familiar with Zstd myself, or its training/dictionary features.

Do you have any idea what happens if a particular dictionary is used during compression, but then unavailable for decompression?

BTW it's worth noting that you can of course pass Nippy's uncompressed output to any compression tool you like. The built in compression schemes are there mostly for convenience.

There's some performance+ratio bench info on the current out-the-box compressors available here in case that's helpful.

Cheers!