search

mholt / archiver

DEPRECATED. Please use mholt/archives instead.
https://github.com/mholt/archives
MIT License
4.45k stars 392 forks source link

sz: Support S2 (close #429) #431

Closed mholt closed 6 days ago

mholt commented 1 week ago

@CompuRoot maybe you can take a look at this. I'll likely just merge this and then we can change it later if needed.

@klauspost -- Thank you for yet again, another great package! (One minor request: It could be useful if the compression levels were exported.)

klauspost commented 1 week ago

@mholt "next version" - meaning the replacement will :)

klauspost commented 1 week ago

For now here is a teaser :)

image

mholt commented 1 week ago

Wow! So, do you think I should wait for that? (Before I make new exported API)

CompuRoot commented 6 days ago

For now here is a teaser :)

And bellow is result of testing a small (140Mb) .vdi file with kopia:

     Compression                Compressed   Throughput   Allocs   Usage
------------------------------------------------------------------------------------------------
  0. s2-parallel-8              62.9 MB      1.5 GB/s     1023     197 MB
  1. s2-default                 62.9 MB      1.1 GB/s     1050     195.9 MB
  2. s2-parallel-4              62.9 MB      1 GB/s       995      187.5 MB
  3. pgzip-best-speed           58.9 MB      816.1 MB/s   1352     214.1 MB
  4. s2-better                  60.6 MB      739.4 MB/s   1019     182.4 MB
  5. pgzip                      56.9 MB      474.1 MB/s   1294     210.9 MB
  6. lz4                        64.7 MB      433.7 MB/s   27       204.1 MB (deprecated)
  7. zstd-fastest               54.4 MB      352.6 MB/s   6311     212.4 MB
  8. deflate-best-speed         58.9 MB      322.3 MB/s   38       135 MB
  9. zstd                       52.4 MB      322.3 MB/s   3266     156.3 MB
 10. deflate-default            56.9 MB      186.6 MB/s   38       135.3 MB
 11. gzip-best-speed            58.4 MB      185.6 MB/s   44       135.4 MB
 12. zstd-better-compression    51.6 MB      108.9 MB/s   3222     176 MB
 13. gzip                       55.9 MB      47.3 MB/s    44       135 MB
 14. zstd-best-compression      47 MB        31.8 MB/s    3367     207.5 MB (deprecated)
 15. pgzip-best-compression     56.1 MB      31.2 MB/s    1299     171.6 MB
 16. deflate-best-compression   56.1 MB      8.6 MB/s     39       135.4 MB
 17. gzip-best-compression      55.9 MB      7.3 MB/s     43       135 MB
klauspost commented 6 days ago

@CompuRoot I am not religious about it, but I have used .s2 for my commandline tool.

The snappy relation is not too important, except for backwards compatibility of reading snappy data.

CompuRoot commented 6 days ago

@CompuRoot I am not religious about it, but I have used .s2 for my commandline tool.

The snappy relation is not too important, except for backwards compatibility of reading snappy data.

You right, if there already exists CLI tool with this extension, then keeping .s2 extension for compatibility is a better solution.

mholt commented 6 days ago

Thanks both for chiming in!

Given that S2 is largely compatible with Snappy, I'm keeping it as part of the Snappy format, with the option of writing Snappy-incompatible data if desired.