Create issue for creating each requested codec

jonstewart commented 3 years ago

A number of new LZ variants have been created and used by operating systems and core services/libraries. They are:

Apple's LZFSE & LZVN, used in macOS
Microsoft's "XPress" family, used in Windows
Facebook's ZStd, used in RocksDB and some Linux filesystems
Google's Snappy, used in LevelDB and other open source libraries
LZ4, a popular open source codec that's very fast
LZO, an older codec that inspired these others and still may be found in usage

Additionally, the 7zip archive tool/format has gained in popularity and now may be used similarly to RAR.

License terms may differ for some of these codecs.

Rather than have a scanner per codec above, it may be possible to conduct a fast one pass scan to detect popular starting points for data in each codec, and then evaluate just those sections of an sbuf.

simsong commented 3 years ago

@jonstewart, please open a separate ticket for each of these. Thanks.

jonstewart commented 3 years ago

LZFSE and LZVN: https://github.com/simsong/bulk_extractor/issues/181

jonstewart commented 3 years ago

Xpress: #187

jonstewart commented 3 years ago

zstd: #188

jonstewart commented 3 years ago

Snappy: #189

jonstewart commented 3 years ago

LZ4: #190

jonstewart commented 3 years ago

LZO: #191

jonstewart commented 3 years ago

And of course how could I forget Brotli?! #192

jonstewart commented 3 years ago

7zip: #193

simsong / bulk_extractor

Create issue for creating each requested codec #156