search

hitman249 / wine-launcher

Wine Launcher - Running Windows games under Linux
http://wine.hostronavt.ru
GNU General Public License v3.0
272 stars 10 forks source link

Look into using zstd for squashfs compression #22

Closed shvchk closed 1 year ago

shvchk commented 3 years ago

https://github.com/hitman249/wine-launcher/blob/eb8d096357cc90b1fcfdd81c9534f9b9d9586e3e/src/src/modules/pack.js#L92

zstd will probably be a better choice due to much better decompression speed, and it can compress almost as good as xz on higher compression levels

> mksquashfs help
...
-comp <comp>            select <comp> compression
...
Compressors available and compressor specific options:
...
        zstd
          -Xcompression-level <compression-level>
                <compression-level> should be 1 .. 22 (default 15)
shvchk commented 3 years ago

Quick comparison of compression speed and ratio with xz on my wine prefix with HoMM 3 (/tmp/x is a ramdisk):

> du -sh wineprefix
723M    wineprefix

> for i in 3 6 9 12 15 18 22; do time mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x; done
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   12,32s user 1,51s system 367% cpu 3,763 total
609M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   55,84s user 1,73s system 374% cpu 15,351 total
605M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   58,90s user 1,77s system 367% cpu 16,506 total
604M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   61,31s user 1,75s system 373% cpu 16,868 total
603M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   77,13s user 1,76s system 307% cpu 25,655 total
603M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   220,34s user 1,56s system 295% cpu 1:15,10 total
598M    /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level $i &>   290,87s user 1,82s system 334% cpu 1:27,62 total
598M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz &> /dev/null  365,15s user 2,01s system 339% cpu 1:48,14 total
589M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% &> /dev/null  366,67s user 1,84s system 342% cpu 1:47,59 total
589M    /tmp/x/x

> time mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% -Xbcj x86 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x
mksquashfs wineprefix /tmp/x/x -b 1M -comp xz -Xdict-size 100% -Xbcj x86 &>   731,24s user 2,21s system 324% cpu 3:46,12 total
588M    /tmp/x/x

Didn't test decompression speed, as I have no doubt zstd is much better at it.

hitman249 commented 3 years ago

1) this command only applies to packaging wine 2) the prefix is not compressed at all 3) in the choice of format, we are limited to the squashfuse utility 4) game data is compressed by this command https://github.com/hitman249/wine-launcher/blob/eb8d096357cc90b1fcfdd81c9534f9b9d9586e3e/src/src/modules/pack.js#L94

shvchk commented 3 years ago
  1. Ah, ok. Might be worth it to look at dwarfs if there would be multiple versions of wine and you expect data to be highly redundant.
  2. Ok, I probably got wrong impression from reading readme
  3. squashfuse seems to support zstd
  4. lz4 is nice, but zstd would probably be even better :smile: In tests below, zstd does its job 10+ times faster while compressing better (even on compression level 1): 
    
> du -sh wineprefix
723M    wineprefix

time mksquashfs wineprefix /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x mksquashfs wineprefix /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null 75,28s user 1,24s system 375% cpu 20,389 total 625M /tmp/x/x

time mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x mksquashfs wineprefix /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> 4,54s user 1,39s system 372% cpu 1,588 total 615M /tmp/x/x

du -sh ~/.config 707M /home/x/.config

time mksquashfs ~/.config /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x mksquashfs ~/.config /tmp/x/x -b 1M -comp lz4 -Xhc &> /dev/null 77,89s user 1,35s system 361% cpu 21,943 total 541M /tmp/x/x

time mksquashfs ~/.config /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> /dev/null; du -h /tmp/x/x; rm -f /tmp/x/x mksquashfs ~/.config /tmp/x/x -b 1M -comp zstd -Xcompression-level 1 &> 5,37s user 1,40s system 353% cpu 1,914 total 537M /tmp/x/x

hitman249 commented 3 years ago

1. dwarfs has an executable too large compared to squashfuse (16 MB+ vs 535 KB) 4. decompression speed is more critical than compression

shvchk commented 3 years ago
  1. Might save much more, but I get your point. In this case zstd is still a good alternative to xz.
  2. Yeah, I guess... lz4hc was ~30% faster at decompression in my in-memory and SSD tests. That compression speed difference though : ) Also, zstd compression ratio is significantly better on higher compression levels. Maybe it's worth to be added as an option some day :wink: