search

AppImage / appimagetool

A low-level tool to generate an AppImage from an existing AppDir
96 stars 15 forks source link

Use default block size with zstd compression #65

Closed Samueru-sama closed 3 months ago

Samueru-sama commented 3 months ago

Indeed this was the problem, I just tested making an appimage with this appimagetool and I no longer have the issue of higher startup time.

image

Fixes #64

Build if someone wants to replicate my benchmarks: https://github.com/Samueru-sama/appimagetool/releases

probonopd commented 3 months ago

How do file sizes compare?

Samueru-sama commented 3 months ago

How do file sizes compare?

In the zen-browser appimage, using the defaults which if I'm not mistaken it uses zstd level 15 and the only difference is the block size:

The 128K block size appimage is 0.3 MiB smaller than the 1MB block size appimage. (80.3 vs 80.6 MiB)

And when setting the compression level to level 1 the difference is 1.4 MiB in favor of the 128K block size appimage.

Worth mentioning that the startup time with the default compression level and 128K block size is about 1.8 Seconds, which is faster than the original gzip even.

probonopd commented 3 months ago

What we would also need to consider: How do the block sizes influence zsync efficiency? (zsync being used for delta updates with AppImageUpdate)

Samueru-sama commented 3 months ago

What we would also need to consider: How do the block sizes influence zsync efficiency? (zsync being used for delta updates with AppImageUpdate)

I don't think there is going to be any significant difference, if at all it will likely be in favor of lower block size since it improves the read time considerably.

And in any case, I think it is far more important for appimages to start faster than a delay when updating, which is a far less common operation than launching the application, the current block size makes appimages take longer to start than they were when using gzip.

probonopd commented 3 months ago

It is a question of how large the updates are, how much traffic we produce. Think of people without high speed internet connection.

Samueru-sama commented 3 months ago

It is a question of how large the updates are, how much traffic we produce. Think of people without high speed internet connection.

Oh sorry, In that case it is likely to be even less of a problem, the resulting appimage itself tends to be slightly smaller as well. 128K block size was the default being used for zstd before, it is also the default used by mksquashfs as far as I know it never had issues with zsync updates.

Is there some data that would suggest having higher block sizes improves zsync efficiency? I think we should always stick to defaults unless proven otherwise.

And btw I live in Venezuela, I lived most of my life having less than 4 mbit dsl internet, I know that pain very well hehe.

probonopd commented 3 months ago

Is there some data that would suggest having higher block sizes improves zsync efficiency? I think we should always stick to defaults unless proven otherwise.

I don't know, but maybe you could do some experiements? This would be very valuable.

The use case is: Think if a big software package, like FreeCAD (many hundreds of MB). Imagine they do nightly builds. Updating to the latest nightly should be only a few MB, because not so much has changed overnight. That is the objective.

Samueru-sama commented 3 months ago

I don't know, but maybe you could do some experiements? This would be very valuable.

I can do benchmarks, but I really don't know how to make the appimages with zsync to do the test on, it seems I would need to host two separate appimages and then measure the bandwidth usage on them.

I think this level of testing is something that should have been done when the block size was changed from 128K to 1M.

probonopd commented 3 months ago

For a somewhat realistic test, one could take, e.g., https://github.com/FreeCAD/FreeCAD-Bundle/releases/tag/weekly-builds, then extract it. First make an AppImage with zsync file of the unchanged AppDir, then change a few files in there (e.g., copy in a few libraries from your host system, overwriting the original ones in the AppDir) and create a new AppImage with zsync file. Then run AppImageUpdate on these.

A local web server (e.g., sudo python3 -m http.server) should do.

These types of tests are important but unfortunately no one never takes the time to do them.

Samueru-sama commented 3 months ago

For a somewhat realistic test, one could take, e.g., https://github.com/FreeCAD/FreeCAD-Bundle/releases/tag/weekly-builds, then extract it. First make an AppImage with zsync file of the unchanged AppDir, then change a few files in there (e.g., copy in a few libraries from your host system, overwriting the original ones in the AppDir) and create a new AppImage with zsync file. Then run AppImageUpdate on these.

A local web server (e.g., sudo python3 -m http.server) should do.

These types of tests are important but unfortunately no one never takes the time to do them.

That's beyond my capabilities unfortunately.

The only way I see it is if I find a project that has a big appimage that already uses this very appimagetool with zsync updates already setup, and I would simply fork the project and modify it to use the appimagetool from my fork and compare the bandwidth usage when updating. But that will be very hard to find as most appimages don't even use the static runtime and zstd compression yet.

Is that testing needed for this to be merged?

probonopd commented 3 months ago

Let's say it would help to get this merged anytime soon.

Samueru-sama commented 3 months ago

Let's say it would help to get this merged anytime soon.

I will interpret that as telling me that I need to carry that unrelated testing.

I think the burden is being shifted here, all that testing of zsync efficiency is something that should have been done when the block size was changed from the default size to 1MB.

Now that I found an actual bug with this change, and it is a significant regression, and the fix is a simple as using the default settings, you are asking me to carry testing on zsync efficiency?

I really don't know how to feel about this probono, hopefully I don't end up having to look for a different implementation of appimagetool that doesn't introduce this regression.

And it is not like I have a choice to go back to using gzip, because it is being dropped as an option as well.

TheAssassin commented 3 months ago

@Samueru-sama we are a really small team with a really limited amount of time. We could not afford sophisticated QA so far, we mostly rely on automated tests and extend them whenever appropriate. Combining multiple tools to perform a test adds some serious complexity even. This is an unfortunate exception to our normal conservative approach of keeping settings as-is. I think reverting to the default block size is a good idea therefore.

I hope I can spend more time again on AppImage, mostly focusing on reviews and security improvements. This should help us prevent such situations.

For the time being, we rely on our community to bring such issues to our attention. They use the tooling every single day and usually notice problems early on. So, thanks to you and all the others for debugging this.

@probonopd please avoid unexpected block size changes to random values in the future. I'm saying random because the values need to be multiples of zsync2's internal fixed blocksize for updates to work efficiently, for instance.

probonopd commented 3 months ago

@TheAssassin I won't make "unexpected" block size changes. I vaguely remembered that there was some relationship between block size and zsync, but the details escaped me. Glad that you have deeper knowledge in this area.

TheAssassin commented 3 months ago

@probonopd AFAIR zsync2's zsyncmake2 variant supports arbitrary block sizes. This means that one could in theory use any block size. But we'll have to experiment with this.