search

AppImage / AppImageKit

Package desktop applications as AppImages that run on common Linux-based operating systems, such as RHEL, CentOS, openSUSE, SLED, Ubuntu, Fedora, debian and derivatives. Join #AppImage on irc.libera.chat
http://appimage.org
Other
8.79k stars 563 forks source link

Use zstd compression #478

Closed probonopd closed 2 years ago

probonopd commented 7 years ago 
Add zstd compression and decompression support to SquashFS. zstd is a
great fit for SquashFS because it can compress at ratios approaching xz,
while decompressing twice as fast as zlib. For SquashFS in particular,
it can decompress as fast as lzo and lz4. It also has the flexibility
to turn down the compression ratio for faster compression times.

The compression benchmark is run on the file tree from the SquashFS archive
found in ubuntu-16.10-desktop-amd64.iso [1]. It uses `mksquashfs` with the
default block size (128 KB) and and various compression algorithms/levels.
xz and zstd are also benchmarked with 256 KB blocks. The decompression
benchmark times how long it takes to `tar` the file tree into `/dev/null`.
See the benchmark file in the upstream zstd source repository located under
`contrib/linux-kernel/squashfs-benchmark.sh` [2] for details.

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

| Method         | Ratio | Compression MB/s | Decompression MB/s |
|----------------|-------|------------------|--------------------|
| gzip           |  2.92 |               15 |                128 |
| lzo            |  2.64 |              9.5 |                217 |
| lz4            |  2.12 |               94 |                218 |
| xz             |  3.43 |              5.5 |                 35 |
| xz 256 KB      |  3.53 |              5.4 |                 40 |
| zstd 1         |  2.71 |               96 |                210 |
| zstd 5         |  2.93 |               69 |                198 |
| zstd 10        |  3.01 |               41 |                225 |
| zstd 15        |  3.13 |             11.4 |                224 |
| zstd 16 256 KB |  3.24 |              8.1 |                210 |

This patch was written by Sean Purcell <me@seanp.xyz>, but I will be
taking over the submission process.

[1] http://releases.ubuntu.com/16.10/
[2] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

zstd source repository: https://github.com/facebook/zstd

Signed-off-by: Sean Purcell <me@seanp.xyz>
Signed-off-by: Nick Terrell <terrelln@fb.com>
Signed-off-by: Chris Mason <clm@fb.com>
Acked-by: Phillip Lougher <phillip@squashfs.org.uk>
develar commented 7 years ago

It would be amazing, because electron-builder uses lzma for windows nsis targets — expected size 35 MB. AppImage uses deflate and size ~51 MB (even more — because 7za is not used, compression is not good (e.g. 7za can compress the same data to 50 MB (1.5 MB difference))).

Yes, we use lzma2 for windows portable and users don't complain that start time is slow, but anyway, for now I decided to not use xz for AppImage because of decompression speed.

Calinou commented 7 years ago

Will AppImages using zstd work on older kernels, such as the ones found in Ubuntu LTS, CentOS or Debian?

probonopd commented 7 years ago

If we would implement zstd, then we would implement it in FUSE, which means it would be independent of the kernel. But we would lose the ability to loop-mount such AppImages using old kernels (is anyone ever doing that, anyway?)

probonopd commented 7 years ago

@develar if it is not too much trouble, could you give the following numbers please?

TheAssassin commented 7 years ago

But we would lose the ability to loop-mount such AppImages using the kernel

We wouldn't lose it, but it won't work on old systems, right? I assume that on newer systems, it should work out of the box, but on older ones, it'd produce an error message.

probonopd commented 7 years ago

Correct. Sorry, I meant "old kernels". Corrected above.

probonopd commented 7 years ago

Zstandard support has been added to squashfs-tools, so if I am not mistaken nothing is stopping us now from using Zstandard for AppImages.

Volunteers for a PR?

probonopd commented 6 years ago 
sudo apt install make gcc

zstd library

git clone https://github.com/facebook/zstd/
cd zstd
sudo make install
cd ..

squashfs-tools

git clone https://github.com/plougher/squashfs-tools
cd squashfs-tools/squashfs-tools
make -j8 ZSTD_SUPPORT=1 GZIP_SUPPORT=0 COMP_DEFAULT=zstd mksquashfs
cd ../..

squashfuse

git clone https://github.com/vasi/squashfuse
...
--with-zstd=PREFIX
probonopd commented 6 years ago

https://github.com/vasi/squashfuse/releases/tag/0.1.101 now contains zstd support.

TheAssassin commented 6 years ago

We should update it in a PR branch, and check whether it builds with that new version.

Ambyjkl commented 4 years ago

Any updates on this issue? I think by this point, zstd support is pretty universal

probonopd commented 4 years ago

Hi @Ambyjkl looks like no one has done any work on this. First step would be to scientifically prove that it actually provides some advantage over what we have in place now.

Ambyjkl commented 4 years ago

@probonopd sorry I kinda forgot about this issue. From my understanding, using zstd would be a one-line change here, right? https://github.com/AppImage/AppImageKit/blob/master/src/appimagetoolnoglib.c#L158

brunoais commented 4 years ago

@Ambyjkl I believe the compression level should be selectable to get a balance between how much space used and how much resources used to compress.

denji commented 4 years ago

https://github.com/mhx/dwarfs

brunoais commented 4 years ago

@denji I suggest you open a new issue with that instead

mgord9518 commented 2 years ago

@denji how does it compare to SquashFS? From what I read it seems to target extremely redundant data, which a lot of AppImages aren't (likely only contain a few text files and sparse data, while precompressed images and binary take up the bulk).

EDIT: after doing a little more reading on it and some of my own tests, DwarFS is really impressive. I got 2x better compression than SquashFS for a given AppImage, both using ZSTD. A big issue is how big it is though, static linking gives >8MB executables which is enormous for this kind of usecase. Compressing the runtime could drop that quite a bit, but it would still require medium to large apps for that weight to break even.

@probonopd ZSTD gets significantly better performance at the same or even better compression (>3x read speed at same compression; ~2.7x at max). This gain is definitely enough to warrant adding support, if not defaulting to it. I can send a PR when I have some extra free time if implementing it is the only issue.

Ambyjkl commented 2 years ago

@mgord9518 thanks for taking time to add this :). I got started with adding zstd support here #1091, you can reference that. Do make sure your code is well-tested or communicate properly that tests will be added in the future (or bad things will happen)

TheAssassin commented 2 years ago

Zstd seems more and more promising nowadays. We should really pick up #1091 again. I'll provide feedback asap.

mgord9518 commented 2 years ago

@TheAssassin

Reading the makefile for AppImageKit's runtime, there's a TODO mentioning splitting apart the builds per each compression type (which would be absolutely necessary with ZSTD considering its relatively larger library size), would the preferred method of achieving this be environment variables?

Eg:

make WITH_ZSTD=1

make WITH_GZIP=1

Etc...

TheAssassin commented 2 years ago

The runtime is built using CMake, not sure what Makefile you mean. https://github.com/AppImage/AppImageKit/blob/master/src/build-runtime.cmake

mgord9518 commented 2 years ago

@TheAssassin

Ah, I was looking at https://github.com/AppImage/AppImageKit/blob/master/Makefile.runtime

TheAssassin commented 2 years ago

I see. No idea why that thing still exists, should be cleaned up some day. Anyway. I think we should just add it with the other compression algorithms, and see what the size difference will be. A few hundred kiB more should not be a problem.

mgord9518 commented 2 years ago

@TheAssassin Based on things I've static linked with ZSTD and backed up by this person's runtime build sizes, it adds somewhere from 100-200 kB, so I'll admit I blew it out of proportion with needing to keep it separate.

TheAssassin commented 2 years ago

That's really an acceptable growth. And we might save some bytes by removing support for other compression algorithms eventually.

mgord9518 commented 2 years ago

@probonopd I haven't tried with the runtime yet, but I have been able to fully statically link squashfuse (which I believe is its biggest dependency?)

probonopd commented 2 years ago

Yes. Completely static, including squashfuse and everything else including libc.

probonopd commented 2 years ago

Discussion continues at

gsantner commented 2 years ago

Is zstd now supported or not?

I create appimages using mksquashfs that supports zstd:

$ mksquashfs
SYNTAX:mksquashfs source1 source2 ...  dest [options] [-e list of exclude
dirs/files]

Filesystem build options:
-comp <comp>        select <comp> compression
            Compressors available:
                gzip (default)
                lzma
                lzo
                lz4
                xz
                zstd

mksquashfs successfully creates the container - so the fuse and squashfs tools support zstd:

mksquashfs $appdir  f.squashfs -root-owned -noappend -comp zstd -Xcompression-level 22
cat f.squashfs >> appimagetool.AppImage

but when I run the appimage it says only xz & zlib is supported.

Squashfs image uses (null) compression, this version supports only xz, zlib.

Cannot mount AppImage, please check your FUSE setup.
You might still be able to extract the contents of this AppImage 
if you run it with the --appimage-extract option. 
See https://github.com/AppImage/AppImageKit/wiki/FUSE 
for more information
open dir error: No such file or directory
probonopd commented 2 years ago

zstd is not yet supported by the normal AppImage runtime, but by this experimental one: https://github.com/probonopd/static-tools/releases/download/continuous/runtime-fuse2-x86_64 (just append your squashfs to it)

gsantner commented 2 years ago

thx I could get it to work with that, and don't notice any (better or worse) performance difference.

Size wise it's sure a nice improvement, ~50MB less to sync to all devices.

291M    retroarch-zstd-22.AppImage
333M    retroarch-gzip.AppImage

some general thoughts, hope it's OK to ask --- and be aware I use AppImage myself, it's good meant critics.

is your runtime/tools project going to replace the current appimage runtime implementation (soon)?

I'm quite heavy user (and packager for my own things) of appimage and I feel the biggest issues/drawbacks is fragmentation - too many tools/repos/things.

Honestly said I use 7z x to unpack appimages, mksquashfs + cat runtime to make AppImages. The rest of the tools & ecosystem feels not trustworthy or something where the tools/arguments gonna work next year still.

I just hope theres more unification in the appimage project and it's tools soon, instead of like 20 repos and 20 tools. Otherwise it's doomed to fail. I'm specifically writing this to you @probonopd as you are the most driving force behind the Appimage project - literally everywhere I looked or found AppImage, you commented already or gave your valuable input.

probonopd commented 2 years ago

Thanks for your feedback @gsantner.

is your runtime/tools project going to replace the current appimage runtime implementation (soon)?

Please watch https://github.com/AppImage/AppImageKit/issues/877 on this.

I'm quite heavy user (and packager for my own things) of appimage and I feel the biggest issues/drawbacks is fragmentation - too many tools/repos/things.

I have to agree. Personally I always wanted a monorepo - one repo that holds the runtime, a tool to make AppImages, and a tool to use AppImages (integrate them into the system) but it turned out that different people had different needs, different wishes, and different favorite programming languages. So it got a bit unwieldy. The plan is to significantly reduce the number of projects within the https://github.com/AppImage/ organization and move everything else out of it.

There are 5 different runtimes, which one is to trust?

Well, there is only one official AppImage runtime, in this repo. And then there are various people's experiments (including mine). Maybe one of the experimental ones will become official sooner or later.

Honestly said I use 7z x to unpack appimages, mksquashfs + cat runtime to make AppImages.

That's a very valid way of doing it if you know exactly what you are doing (i.e., what to bundle and how).

Various people have made various tools (and counting). I don't think that's a bad thing per se but it can be a bit overwhelming at times.

I just hope theres more unification in the appimage project and it's tools soon, instead of like 20 repos and 20 tools.

How would you go about achieving this?

gsantner commented 2 years ago

How would you go about achieving this?

Starting at runtime - you are pretty avid on the new static-tools runtime development. Why not make it a branch at AppImagekit and call it the AppImageKit next version alpha?

As of monorepo, definitely all sides input is important - users, maintainers, developers and packagers alike. What I for example don't get is why the updating functionality is split out. I would prefer it as a general opt-in feature of the runtime, controlled by config files inside the appimage.

Then the website, general info and "where to find appimages". It really shouldn't need~10 places to get informed, and then again one or other place is outdated, missed info of another..sync issues. And ~10 places to maintain also means less time for adding actual content - so writing down where to download i.e. application X as appimage.
I tried a varity of appimage "finder" utility and pages, but at the end of the day the best way to find what I need is to specifcally look for a project on git hosts and searching for appimage. Like where could I find LibreWolf (firefox)? None! :smile: .

I guess a unified website containing information, recipes and list of application links would be better than spreading it all over the place. Like a single own Jekyll/Hugo repository for everything, and users can add a new application by making a pull request to that repo, mapping them i.e. with a json file.

probonopd commented 2 years ago

@gsantner what you are suggesting is roughly along the lines of what @TheAssassin and I had brainstormed recently. So I think we should turn it into action soon-ish. By the way, we are always looking for volunteers. :-)

Like where could I find LibreWolf (firefox)?

appimage.github.io lists AppImages that have passed the automated test on the oldest still-supported LTS release of Ubuntu. I tried to add LibreWolf three times, but all tests so far have failed:

OK, it's now 2022 so maybe I should give it another try...

...and it seems like the test finally succeeded :+1:

https://appimage.github.io/LibreWolf/

probonopd commented 2 years ago

don't notice any (better or worse) performance difference

Maybe, if you have some time, you could benchmark this with different parameters for block sizes and compression levels? And document the result, so that we can pick ideal defaults in upcoming tools.

Samueru-sama commented 9 months ago

Hi, I would like to recompress some appimages I have to zstd to improve the execution time, I did some tests by manually compressing the .AppDir to gzip vs zstd:1 and the difference is quite significant in the decompression speed: 

tar -xzf Brave.AppDIr.tar.gz  2.90s user 0.75s system 116% cpu 3.127 total
tar -I zstd -xf Brave.AppDIr.tar.zst  0.63s user 0.58s system 202% cpu 0.599 total

If I try to use appimage tool it tells me that "Only gzip (faster execution, larger files) and xz (slower execution, smaller files) compression is supported at the moment".

Sorry is this isn't the right place to ask this.

probonopd commented 9 months ago

Please try the next-generation https://github.com/AppImage/appimagetool/ - does it work with that one?

Samueru-sama commented 9 months ago

Please try the next-generation https://github.com/AppImage/appimagetool/ - does it work with that one?

Thanks, this one works. Here are some benchmarks:

The new Brave zstd appimage is 150 MB while the original appimage is 161 MB.

Original:

./Brave.AppImage --appimage-extract > /dev/null  1.84s user 0.29s system 85% cpu 2.499 total
./Brave.AppImage --appimage-extract > /dev/null  1.77s user 0.30s system 84% cpu 2.440 total

Zstd:

./Bravezstd.AppImage --appimage-extract  0.64s user 0.30s system 99% cpu 0.936 total
./Bravezstd.AppImage --appimage-extract  0.60s user 0.38s system 99% cpu 0.980 total

The extraction time is significantly faster.

(Something interesting is that doing a appimage-extract to the zstd appimage doesn't result in the list of files being printed on the terminal, but doing it to the original appimage does print that info).

And here are benchmarks of the startup time of the appimage:

Original AppImage: 

Time taken: 2.49 seconds
Time taken: 2.69 seconds
Time taken: 2.70 seconds

Zstd appimage: 

Time taken: 4.19 seconds
Time taken: 4.19 seconds
Time taken: 4.15 seconds

Huh, that's weird. It takes longer to start with zstd. How can I find which compression algo is being used on the original appimage? And do I have to do something other than ./appimagetool-x86_64.AppImage --comp zstd Brave.AppDir when creating zstd appimage that improves the startup time?

Also what's the default compression level when using zstd and can it be changed?

probonopd commented 9 months ago

That's weird indeed. I mean, we picked zstandard because it should be faster at runtime... unfortunately I don't know all the answers, more investigation is welcome. At least the algorithm, block size, and compression level play into this. We have to make tradeoffs between image size, runtime speed, and zsync efficiency. To find out the compression algorithm of the first AppImage, you might run it with --appimage-offset and then remove that number of bytes from the beginning of the file (in other words, make a copy of the file skipping that number of files). This way, you get the "pure" squashfs file, which you can run the file command on. With some luck it should show you the compression algorithm. Maybe running unsquashfs on it shows even more information, I am not sure whether it shows the compression level and block size, though.

Samueru-sama commented 9 months ago

That's weird indeed. I mean, we picked zstandard because it should be faster at runtime... unfortunately I don't know all the answers, more investigation is welcome. At least the algorithm, block size, and compression level play into this. We have to make tradeoffs between image size, runtime speed, and zsync efficiency. To find out the compression algorithm of the first AppImage, you might run it with --appimage-offset and then remove that number of bytes from the beginning of the file (in other words, make a copy of the file skipping that number of files). This way, you get the "pure" squashfs file, which you can run the file command on. With some luck it should show you the compression algorithm. Maybe running unsquashfs on it shows even more information, I am not sure whether it shows the compression level and block size, though.

I just checked the source of the appimage, they use gzip: https://github.com/srevinsaju/Brave-AppImage/blob/master/.github/workflows/release.yml

./appimagetool.AppDir/AppRun --comp gzip "$APPDIR" -n -u

I just went and recreated the steps, this time recompressing the extracted dir to gzip with the older version of appimagetool I have (the newer one tells me that it can only do zstd) and now it took 2.7s to start. So that rules out an issue with recreating the appimage.

If you want you can try to replicate my results, I extracted it, renamed it to Brave.AppDir and used appimage tool to turn it into a zstd appimage and compare the startup time of both.

What's the default zstd compression level being used in appimagetool? If it is something like 9 maybe that is what causes the increased delay. Lots of testing and benchmark I've seen indicate that using anything higher than 2 is rarely needed, it causes something like a 20% decrease in decompression speed for a 2% gain in compression ratio.

probonopd commented 9 months ago

Yes, we have never really systematically analyzed the optimal compression levels.

Looks like we are currently invoking mksquashfs without any particular compression level, so whatever it uses by default gets used:

https://github.com/AppImage/appimagetool/blob/bfe6e0c1c663b6f58c0759d942abc3a6d1729c75/src/appimagetool.c#L156-L164

Maybe you'd like to add a different compression level there and compare the results. Thanks for your contribution!

Samueru-sama commented 9 months ago

Yes, we have never really systematically analyzed the optimal compression levels.

Looks like we are currently invoking mksquashfs without any particular compression level, so whatever it uses by default gets used:

https://github.com/AppImage/appimagetool/blob/bfe6e0c1c663b6f58c0759d942abc3a6d1729c75/src/appimagetool.c#L156-L164

Maybe you'd like to add a different compression level there and compare the results. Thanks for your contribution!

If it is using the default it means that it is using zstd:3. The difference between 1 and 3 isn't that big that I would think that what causes the near extra 2 second delay. But it would need to be tested either way.

Thanks for pointing out what to edit in the code of appimage tool, unfortunately that is beyond my knowledge at this point, the build instructions are for docker (which I don't know how to use) and there isn't an aur appimagetool-git package that I could use as reference to make it.

I might test other appimages gzip vs zstd to see if it still takes longer to start with those.

EDIT: Doing the same test with the librewolf appimage.

Original startup time: 

Time taken: 2.70 seconds
Time taken: 2.81 seconds
Time taken: 2.70 seconds

Zstd: 

Time taken: 3.79 seconds
Time taken: 3.57 seconds
Time taken: 3.67 seconds

It is also slower with the default zstd compression.

JulianGro commented 9 months ago

I wonder if a bigger AppImage, like Overte (>500MiB) would be an interesting benchmark. https://overte.org/downloads.html Though, like you said, the ~2 second delay seems fishy. Zstd is way faster than gzip, at least that is my experience when using them as BTRFS filesystem compression.

mgord9518 commented 9 months ago

@JulianGro Depends on implementation. Using libdeflate closes the gap but ZSTD is still faster. Libdeflate is 2x as fast as zlib, while zstd is 3x as fast for normal usage.

probonopd commented 9 months ago

Thanks for pointing out what to edit in the code of appimage tool, unfortunately that is beyond my knowledge at this point, the build instructions are for docker

Alright, I see. A much easier and quicker way to do some tests would be to

  1. Extract an example AppImage, e.g., the LibreWolf one
  2. Using mksquashfs, make different images out of it with different algorithms, compression levels, block sizes
  3. Download https://github.com/AppImage/type2-runtime/releases/download/continuous/runtime-x86_64
  4. Using cat, create AppImage files by combining the runtime with the squashfs file (just append the squashfs to the runtime)
  5. Benchmark the resulting AppImages

Please let me know if you'd like me to elaborate on how exactly to do that.

Samueru-sama commented 9 months ago

Thanks for pointing out what to edit in the code of appimage tool, unfortunately that is beyond my knowledge at this point, the build instructions are for docker

Alright, I see. A much easier and quicker way to do some tests would be to

  1. Extract an example AppImage, e.g., the LibreWolf one
  2. Using mksquashfs, make different images out of it with different algorithms, compression levels, block sizes
  3. Download https://github.com/AppImage/type2-runtime/releases/download/continuous/runtime-x86_64
  4. Using cat, create AppImage files by combining the runtime with the squashfs file (just append the squashfs to the runtime)
  5. Benchmark the resulting AppImages

Please let me know if you'd like me to elaborate on how exactly to do that.

Alright I think I got it. The only thing I'm not sure if I did right was the append part with cat, I did this with the zstd1 squashfs image:

cat zstd1.squashfs >> runtime-x86_64

And now runtime became the appimage, its size increased to match the size of the squashfs image and it launches librewolf when run so I assume I got it right.

Here are the results for for several appimages, I made several copies of the runtime and appended different squashfs images with different comp levels to each and proceeded to benchmark the startup time of each, I only changed the compression level, didn't change anything else:

zstd3 (which is 118MB): 

Time taken: 1.82 seconds
Time taken: 1.93 seconds
Time taken: 1.93 seconds

zstd1 (125MB): 

Time taken: 1.82 seconds
Time taken: 1.82 seconds
Time taken: 1.93 seconds

zstd6 (110MB): 

Time taken: 1.93 seconds
Time taken: 2.04 seconds
Time taken: 2.04 seconds

And for reference, here is the startup time of the original librewolf appimage (likely gpzip) it is 112MB: 

Time taken: 2.36 seconds
Time taken: 2.9 seconds
Time taken: 2.58 seconds

This is good, even zstd6 is faster by a considerable amount!

And here is the startup time of librewolf zstd appimage, created using ./appimagetool-x86_64.AppImage --comp zstd librewolf.AppDir

I think appimagetool is doing more than just using the default zstd 3 compression level because the resulting appimage is 97 MB in size?! That's indicative that is using a very high compression level. 

Time taken: 3.43 seconds
Time taken: 3.43 seconds
Time taken: 3.64 seconds

That explains everything. There is something going on with the appimagetool that causes the very compressed appimage, which on one end is good because it compressed it considerable, however now the application is also considerably slower on startup.

Maybe for application like web-browsers the documentation should state to not go too hard on the compression level, already zstd1 is smaller than what a native package of librewolf would be (340 MB) so I only see downsides with using a too high compression level for this type of application.

Thanks for the help probono, I hope I did all the steps correctly.

Here is how I'm benchmarking the startup time, it is a launcher script that launches the appimage in the same directory and will stop counting once the i3wm window class matches the one of the appimage which for librewolf is 'LibreWolf' (I have i3 configured to automatically focus on new windows once they spawn). 

#!/bin/bash

# Get the directory of the script
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

# Start the application (replace 'APPLICATION' with the actual application name)
"$DIR/APPLICATION" &

# Get the start time in nanoseconds
start_time=$(date +%s%N)

# Loop until the window class matches "WINDOWCLASSHERE"
while true; do
    window_class=$(i3-msg -t get_tree | jq -r ".. | select(.focused? == true) | .window_properties.class")
    if [[ "$window_class" == "WINDOWCLASSHERE" ]]; then
        break
    fi
    # Sleep for a bit to prevent high CPU usage
    sleep 0.1
done

# Get the end time in nanoseconds
end_time=$(date +%s%N)

# Calculate the time taken and convert to seconds
time_taken=$(echo "scale=2; ($end_time - $start_time) / 1000000000" | bc)

# Output the time it took to run
echo "Time taken: $time_taken seconds"

The benchmark script was written by AI, so no idea how terrible it might be, but it has worked well so far haha.

Edit: I also tested brave with the mksquashfs and append method, the startup time is 1.63 seconds when using zstd1, very good!

brunoais commented 9 months ago

The benchmark script was written by AI, so no idea how terrible it might be, but it has worked well so far haha.

The main issue this has is its 0.1 sleep which means that the numbers you show are only reliable to a between -0.2 to 0 seconds. The 2nd decimal digit has no reliable value and the first decimal digit is so-so.

As long as you stay in the realm of the 0.5s, it's ok

mralusw commented 7 months ago

@Samueru-sama @probonopd the mksquashfs default is zstd:15. Nothing to do with appimagetool.

Luckily it's simple enough to change the level, by passing appimagetool the --mksquashfs-opt flag twice (because mksquashfs doesn't take -Xcompression-level=N — it insists on two separate CLI args; hey, what did you expect from a tool that seems bent on having inconvenient usage?):

appimagetool --comp zstd --mksquashfs-opt -Xcompression-level --mksquashfs-opt 3 ...
mralusw commented 7 months ago

@probonopd if you want to implement different compression levels, I'd suggest accepting "zstd:N" (just as mount.btrfs accepts). There are tools that call appimagetool internally (linuxdeploy), and an extra level of CLI arg wrapping gets a bit insane.

probonopd commented 7 months ago

Well, ideally we find the optimal ("balanced") settings and set them for everyone. So that we make the choice, and not every application developer has to think about which compression level to use.