Statically linked runtime

[agowa]

AppImages should run on all Linux Platforms, but currently they don't, this is because it is dynamically linked against glibc. I tried to run a AppImage on Alpine Linux and it failed because Alpine Linux is build around musl libc instead. I think AppImages should generally include all necessary dependencies and not some of them. Also adding libc would not increase the resulting size much, depending on the used libc, it may only be from 185k to 8M libc Comparison Chart. And if the binary is also stripped it can also be a much less.

AppImage should do something like this:

1. When creating, check the required symbols of the application and determine the smallest fitting libc to include.
2. Always produce statically linked binaries, that don't depend on anything on the system.
3. Recommend using musl libc instead of glibc, for various reasons like License (MIT ./. LGPL), binary size (527k ./. 8M).

[TheAssassin]

Linking just libfuse3 statically already increases the binary size, though... although I can't say I tried. And you'd need a statically linked libfuse3 to begin with.

[s09bQ5]

Statically linking libfuse3 adds around 180kB. Compiling libfuse3 with -ffunction-sections and -fdata-sections reduces this to 98kB.

Another reason for the large size difference between https://github.com/AppImage/AppImageKit/releases/download/continuous/runtime-x86_64 and https://github.com/probonopd/static-tools/releases/download/continuous/runtime-fuse2-x86_64 is the switch from lzma to zstd. Even when I compiled libzstd with -ffunction-sections and -fdata-sections the runtime was 143kB larger than with liblzma. Statically linking zlib adds just 36kB.

[bksubhuti]

should not crash anything. The only issue is that Ubuntu does no longer ship it by default, like it did for decades. And we did not get a couple of years to transition from libfuse2 to libfuse3. They did not ship both in parallel for a few years as would have been the thing to do (in my opinion).

If you mistakenly run the command on 23.04 (I think also 22.04), and you reboot, you will crash your system.. trust me.. or try it yourself. sudo apt install fuse The old-school way was to do it that way.. and that is the problem.

As for size.. If you have an appimage that is 1 mb with a GUI.. I'd be far fetched to see it. Usually, apps are 5mb minimum.. maybe huge.. Ours is between 110-120mb because of a 100 mb sqlite db. We are not worried about size.

Why not use something other than fuse? Why not just have different options during the AppImage process so everyone is happy?

[probonopd]

Why not use something other than fuse?

Because FUSE is the only way we know to mount a disk image without needing root rights.

[probonopd]

@s09bQ5 most AppImages are tens to hundreds of MB, do you really think we should be worried about <1MB size overhead? In any case, will apply your suggested size optimizations, which squeeze out another ~3%.

[mgord9518]

@bksubhuti FUSE is the only way to mount things without needing root on Linux. The only other options would be requiring root or fully extracting to a cache directory.

A different implementation of libFUSE (the C interface to FUSE) however, is possible. Unfortunately, this actually still requires libFUSE to be installed on the host system due to 'fusermount' needing to be suid.

The only realistic option I can think of would be to fork libFUSE3, strip it down to the only parts needed for AppImage and hope that libFUSE doesn't break fusermount compatibility too often

[bksubhuti]

I hope something can be done with a custom fuse and rename the public exposed functions to avoid conflict. I'm very happy with my extract, rename, zip and ship method. It too does not need root and "it just works" and it is portable like appimages. So I'm happy. You can see the app here which is quite big and involved.. need libsqlite3-dev https://americanmonk.org/tipitaka-pali-reader/

[bksubhuti]

If it means anything.. there is a conversation over here.. look at the last 2 comments you see someone talking about fuse2 and my comments. I think there is a plot to destroy appimages in favor of snap and flatpak. I think you better act quickly. In the conversation, the user already switched all of his appimages to flatpak. Here is the conversation: https://github.com/bksubhuti/tipitaka-pali-reader/issues/173#issuecomment-1605829757

Furthermore, I cannot get appimages to on Lubuntu 22.04 with libfuse2 installed. It says permission denied if I launch from the command line. Yet if I extract and run, it works fine. You will surely hear more of more trouble from others (who care to comment) as time passes. Remember that 1/100 people will comment.

[probonopd]

The only realistic option I can think of would be to fork libFUSE3, strip it down to the only parts needed for AppImage and hope that libFUSE doesn't break fusermount compatibility too often

That's basically what we are doing by linking libfuse statically, which means that only the functions that we are actually using will become part of the runtime.

[mgord9518]

@probonopd is it an entirely new codebase or a hard fork of the old version? And what's the reasoning behind another new one?

I think it should use the static runtime, or use some simple mechanism to detect whether the app being built requires glibc and using a dynamic linked runtime if so.

[probonopd]

@mgord9518 it's based on the existing codebase but will increasingly replace glib with C++ standard library calls, and is statically linked with musl libc on Alpine Linux for increased binary compatibility without having to build on old systems.

[TheAssassin]

What do people think, should it use the new static runtime by default, or should it be only opt-in?

I oppose any such polls. Most people do not have the necessary background knowledge or know the context sufficiently to properly vote on such a topic. This is pretty much a dishonest attempt to prove your point. Do you really need to resort to such measures?

After all, https://github.com/AppImage/appimagetool/pull/33 already cites a prior debate with you and ultimately implements your requirement of not making major changes before a stable release unless the result is essentially compatible with the "old" appimagetool from https://github.com/AppImage/AppImageKit.

[TheAssassin]

I think it makes sense to use the static runtime by default if it has feature parity with the old runtime.

See, this is exactly what I meant. This is not really what @probonopd asked for. There is a consensus of moving forward with a fully static runtime. @Adam- was led to spending time on creating a response that ultimately confirms a decision that was already made.

Folks, it's not as easy as "switching it on or switching it off". The new runtime creates major problems for community software like https://github.com/TheAssassin/AppImageLauncher. A solution is in progress. Meanwhile, no appimagetool should create AppImages that may break on the systems of a significant share of users without any error message or any kind of feedback and with a very cryptic error message.

[mgord9518]

@TheAssassin Why not a flag to enable the static runtime and package both of them? There needs to be some way to test the new runtime without going way out of the way and downloading a different tool (like go-appimage). Something experimental will never become stable if there's no easy way for the community to test it

[TheAssassin]

@mgord9518 it's not a fork, it's basically moving the old code base into a standalone repository to make it easier to manage it. Being the official (future) successor and the new home of appimagetool, all changes we had to make (or wanted to) went into that repository.

At this point, it's not the "official", recommended tool to use. I see it as a tech preview we have not announced anywhere as of yet.

package both of them?

We decided to no longer ship runtimes, but automatically download them, as you can see in https://github.com/AppImage/appimagetool/pull/33. (Edit: this prevents cross dependencies to releases in different repositories, basically, and ensures people have the latest runtime by default, which I think is a great addition!) I initially thought using the static runtime by default was the right thing to do at this point already, but using it from https://github.com/linuxdeploy/linuxdeploy-plugin-appimage as a kind of field test proved to be a really, really bad idea. I am in contact with software companies which ship AppImages, and the resulting AppImages for some tech preview suddenly broke on customer PCs with a cryptic error message only visible in the terminal. Not a very good UX. The reason is that the static runtime is still incompatible with AppImageLauncher (see https://github.com/AppImage/AppImageSpec/issues/38), which they recommend to their users.

I am convinced that it is a good idea to use the new appimagetool and again want to use it in linuxdeploy-plugin-appimage. But I cannot ship software which generates AppImages that currently break in many cases.

Also, I'm not saying it's the runtime's fault. It's the spec which we need to adapt.

Note: there is a reason neither https://github.com/AppImage/appimagetool nor the new static runtime have been officially released and that this issue is still open. The former is a tech preview on its way to its first release candidate. The latter needs some engineering for its build system and some nice continuous integration testing (the code itself seems good enough for now, I just recently found some minor memory issues lately that also affect the old runtime code in AppImageKit).

[TheAssassin]

Please note that I'd immediately change to the new type2-runtime repository once that all the issues have been resolved and we ran some tests in common, real-world scenarios, ideally with you, the community. I see that everyone is really keen on making the change sooner than later. But the world of software is the world of technical debt. Let's fix the problems we already know about.

[bksubhuti]

A user told me that Libefuae2 is being automatically removed when people upgrade their system. I think this was popOs. That user "upgraded" all apps to flatpak. I personally have a flatpak PR in progress. Flatpaks are not fun to build.
Appimages should "just work". I myself, and one other user crashed my whole system by installing fuse, which by default installs fuse3. Install Ubuntu 23.04. And install "fuse" without the number. Then try to reboot. You will not be able to do so. The old instructions were to install fuse. Fuse2 is a special instruction. Those who insist on lean apps should decide. What is the price.? I think if fuse2 is forked and cherry picked there is maybe 100KB and 700 KB if not. GUI apps are no affected by this.size. The big question was who upgraded fuse2 and why did they upgrade to the point where it crashes Ubuntu, the most widely used base for most distros. (In user numbers). I suspect the snap flatpak distribution war. Ubuntu refuses to certify distros that have flatpak . Most people don't like 2 gb flatpak Whatsapp installs. What is 100 or 700KB compared to that.?

[TheAssassin]

@bksubhuti FUSE is the only way to mount things without needing root on Linux. The only other options would be requiring root or fully extracting to a cache directory.

@mgord9518 I think you can do fancy stuff with cgroups. Browsers (including Electron's runtime), tools like podman and buildah and many more examples show you can perform mounts. We may be able to get rid of FUSE in the future. I'm not an expert on cgroups, though, and I'm relatively sure the solution would add some complexity over just using FUSE (where we don't even need to worry about the filesystem, as this software already exists).

I think we should open an issue for that.

[bksubhuti]

Fuse should just work. It should also prompt to install desktop files and copy to a ~ /local share etc folder on first use.

Users would rather install 2gb flatpaks than 30 MB appimages for this feature.

You still have not mentioned what the price is for static linked fuse2

What is the drawback besides "Just Working". ?

On Mon, Jul 17, 2023, 05:09 TheAssassin @.***> wrote:

@bksubhuti https://github.com/bksubhuti FUSE is the only way to mount things without needing root on Linux. The only other options would be requiring root or fully extracting to a cache directory.

@mgord9518 https://github.com/mgord9518 I think you can do fancy stuff with cgroups. Browsers (including Electron's runtime), tools like podman and buildah and many more examples show you can perform mounts. We may be able to get rid of FUSE in the future. I'm not an expert on cgroups, though, and I'm relatively sure the solution would add some complexity over just using FUSE (where we don't even need to worry about the filesystem, as this software already exists).

I think we should open an issue for that.

— Reply to this email directly, view it on GitHub https://github.com/AppImage/AppImageKit/issues/877#issuecomment-1637217703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKXQWEX3YNXSYFA267KRWLXQR3R7ANCNFSM4F7UMLVA . You are receiving this because you were mentioned.Message ID: @.***>

[TheAssassin]

You seemingly want to continue a unilateral off-topic discussion based on a strawman fallacy. Nobody is against using a static runtime. It's a matter of "how do we implement this without breaking other stuff", i.e., everyday business for software engineering.

For questions on this proposal I just made, please use the issue that is linked just above your comment. I strongly recommend you use GitHub's web interface.

[bksubhuti]

What is the price to pay for static linked lib.? Originally it was 700KB. But you mentioned pulling only necessary items and reducing that.

Some would argue that static will not break whole systems like fuse3 does

[TheAssassin]

Please stop posting off-topic messages. This has been answered in detail in the discussion above.

[bksubhuti]

You could just have a newer differently named appimage maker and not worry . Have it in beta for a year and see what happens.. I have a small user base and can test

[TheAssassin]

This might be a language barrier issue...

[mgord9518]

@TheAssassin That (using cgroups to mount without root) sounds pretty interesting as a fallback. Assuming the implementation isn't super complex, I wonder if it would be feasible to write a wrapper around it to make the API identical to libFUSE, which could be maintained as a separate project.

I'd be happy to do some research, my current knowledge of cgroups is essentially nil but it's cool to hear there are other options

[bksubhuti]

I am a native speaker of Merikan English but I mostly live abroad. I just lost track of the issue and previous history.. My question is, is the information below actually the idea where "just pieces were pulled and compiled directly" or if the whole fuse2 library was statically linked?

Here it is.

Go? how large is the final static runtime?

Depends on the architecture, around 2 MB: https://github.com/kost/static-appimage/releases

When you run upx -9 on it, you can bring it to under 1 MB.

[bksubhuti]

I also looked at the static release folder and noticed there are releases for each linux OS. Is this for the end user as well or just the user who builds.

I'd be happy to try using it, but I don't know what to do. I'm not so familiar and intuitive with today's assumed knowledge. If you give me a command or a series of commands I can run on an ubuntu 18.04 droplet. I'd be happy to test and also ask a few others to test my app. Our flutter app is quite heavy and uses sqlite3 as well.

my commands to build are below: https://github.com/bksubhuti/tipitaka-pali-reader/blob/master/serverbuildcommands.sh Let me know what to do and I will run it.

[mgord9518]

@bksubhuti the repo you listed is not the runtime being discussed here. The static runtime being talked about here is just the original runtime being linked statically instead of dynamically (at least to the best of my knowledge)

[xplshn]

Just to clarify; When you statically link something, it only uses the symbols relevant to the program, sometimes it is even smaller than using dynamically linked binaries, because a dynamically linked binary makes the LD lookup where the libraries are, which means;

• Increases attack surface, I could change $LD_LIBRARY_PATH and put a library there of the same name and behavior but malicious. Also, if a library, like, let's say xz-libs (which required systemd and glibc to be exploitable...) updates, all programs based on it are vulnerable until xz-libs is fixed, this is both a cons and pro for dynamically linking.
• No versioning: When Glibc updates, programs compiled against it inevitably break. Or you'd have to have a versioned glibc, which in its vanilla form takes 50MB as is... So, for each version you'd have +50MB of .so ELFs.
• Load time - If program A depends on /usr/lib/libpipewire-0.3.so.0, and /usr/lib/libpipewire-0.3.so.0 depends on other libraries, which in turn depend on some other libraries, the chain gets bigger until it starts to noticeably slow down the startup, also, that means that memory is loaded in a kind of asynchronous, and thus irregular way (not a problem with today's tech, but still).
• Invisible size, your binary may be 4megs but the libraries its loading, and the libraries that each library loads could very well be bigger than 30 megs...

There are LOTS of drawbacks, but those are the most important, read this if interested: https://harmful.cat-v.org/software/dynamic-linking/

I see one possible "fix" for now... AppImages could detect missing libraries in the system prior to running, then pull such libraries from a repo containing the missing libraries compiled against Musl, uClibc or even Glibc if need be, so long as they are made portable appropriately, also, note that using Patchelf one can modify ELFs and change where they look for dependencies or you can also replace their hard dependencies or remove them so that the Dynamic Loader lets you run the program even without loading such dependency. If a repo like that is to ever be created, it could be a simple Github Actions powered repo compiling libraries against various libcs. Flatpaks resolved this issue by having different runtimes, each with glibc included (glibc is the root of all evils, that's why most runtimes weight more than 100megs, because they are compiled against glibc).

:)

[s09bQ5]

@xplshn what you talk about sounds very off-topic. This issue is about the application that provides the filesystem for the real application inside the AppImage. The dependencies of the application inside the AppImage are beyond its scope.

There is no increased attack surface due to dynamic linking. If you can run code to change LD_LIBRARAY_PATH, you don't need to use this to persuade another application to run your malicious code. You can run your malicious code directly. When application gain privileges through setuid LD_LIBRARY_PATH is ignored.

Glibc is the prime example of backward compatibility. It has stuck to the same soname for the past 27 years and provides multiple versions of a symbol to support applications that need the old behavior of a function.

Your points about load time and invisible size might also be false. If an application loads libpipewire from outside the AppImage, there is most likely already another application (e.g. pipewire itself) that has this library loaded and the Linux kernel will share pages between these application as long as they remain unchanged. The dynamic loader just has to resolve the symbols, which is usually done lazily.

[xplshn]

Glibc is the prime example of backward compatibility. It has stuck to the same soname for the past 27 years and provides multiple versions of a symbol to support applications that need the old behavior of a function.

Try running something compiled in Ubuntu 10.4 in Ubuntu 22...

I cited that webpage because it'd be bad seen if I pasted here a 100 lines long explanation and demonstration of the pitfalls of dynamic linking. If you are interested in learning, read this.

[xplshn]

You could also read this very informative paper: http://www.nth-dimension.org.uk/pub/BTL.pdf - Title: "Breaking the links: Exploiting the linker"

[xplshn]

https://catonmat.net/ldd-arbitrary-code-execution

[s09bQ5]

Try running something compiled in Ubuntu 10.4 in Ubuntu 22...

Tried hdparm, works.

I cited that webpage because it'd be bad seen if I pasted here a 100 lines long explanation and demonstration of the pitfalls of dynamic linking. If you are interested in learning, read this.

Learning from that page? That's just a collection of anecdotes from ancient UNIX systems.

You could also read this very informative paper: http://www.nth-dimension.org.uk/pub/BTL.pdf - Title: "Breaking the links: Exploiting the linker"

How do the quirks described in there affect AppImages? The only takeaway I see is that when setting LD_LIBRARY_PATH in an AppImage, one should add : only if the variable is not empty.

https://catonmat.net/ldd-arbitrary-code-execution

This is an attack on people trying to analyze a binary with ldd, not on people who intend to run the binary.

[xplshn]

How it relates to AppImages? Binaries compiled statically are portable when packaged as an AppImage, dynamically linked ones are not, and not only are they not, their pitfalls outweigh the benefits.