New archivers support: Brotli, LZ4 and ZStd

[stokito]

Recently was published few new compression algorithms and formats that are going to be quite popular: Brotli from Google: is supported by all browsers, was standardized and have it's RFC but it doesn't have magic signature. LZ4 which is ultra fast and lightweight. ZStandard from Facebook which is already widely supported by a lot of systems including Linux kernel. It's mime type is application/zstd and it's magic is \x28\xB5\x2F\xFD Could you add a support for them? If yes please also add their tar version (tzst, tlz4, tbr). The more programs supports them then the easier it will be to migrate to this compressors.

[cdgriffith]

Thanks for bringing up these new formats, if you happen to have any of them please post example files.

[stokito]

well, on linux you can easily install the programs and create the test files yourself:

$ sudo apt install brotli lz4 zstd
$ touch test.txt
$ brotli -k test.txt
$ lz4 -k test.txt
$ zstd -k test.txt

I uploaded results:

• http://stokito.com/test.txt.br
• http://stokito.com/test.txt.lz4
• http://stokito.com/test.txt.zst Please download them until my site is working.

Magics:

• Brotli: doesn't have a magic number. We can only determine it by the file extension .br.
• LZ4: 4 Bytes, Little endian format. Value : 0x184D2204 https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md
• ZStandard: Little endian 0xFD2FB528

Mime types:

• application/x-brotli and application/x-brotli-compressed-tar for tar.br files
• application/x-lz4 and application/x-lz4-compressed-tar for tar.lz4 or tlz4 files
• application/zstd (without the -x) and application/x-zstd-compressed-tar for tar.zst or tzst files

[cdgriffith]

Added brotil magic number in 1.7 for format 3 from https://github.com/google/brotli/issues/298 lz4 and zst added in 1.6