Add secondary, more detailed version marker and specify `TARGET_APPIMAGE`

[TheAssassin]

While experimenting with fully static type-2 runtime builds, we found that this breaks AppImageLauncher's integration method which currently depends on LD_PRELOAD. I recently discussed this problem with @probonopd, and we came up with a solution.

Some background: AppImageLauncher depends on binfmt_misc for its core UX feature. All calls to (executable) AppImages are intercepted by AppImageLauncher. Of course, it cannot "just launch" the AppImage any more, resulting in a loop. It needs to bypass binfmt_misc therefore to launch the AppImages after treating them.

To do so, currently, AppImageLauncher extracts the runtime into an in-memory buffer and there, it patches out the magic bytes. This allows it to launch the binary normally while minimizing RAM consumption and avoiding having to copy the entire AppImage to a temporary location. To my own surprise (at least to some extent), this works extremely reliably with the classic upstream AppImageKit type-2 runtime and also the good old type 1 runtime.

Of course, AppImageLauncher needs to tell the runtime where the AppImage really is. This is done by hooking into some of the functions we know it uses that, e.g., open a file handle on the AppImage file on the disk. There, the path can be patched dynamically.

In the past, in the very first versions, TARGET_APPIMAGE even was the main method of telling the runtime which AppImage to launch. Back then AppImageLauncher shipped its own upstream type-2 runtime, which limited the user experience quite a bit. The flag would work with the method described above, though. There are multiple reasons I no longer use it, though. First of all, this feature never made it into the spec, which means that only the upstream runtime can work with it and all other AppImages would crash. We cannot safely assume that all type-2 AppImages provide it. Even if just 10% of the AppImages would be incompatible, they would just crash. The "hook into them with LD_PRELOAD method" is not affected by this limitation. We have no means of distinguishing between those runtimes.

To make the operation more reliable and allow the tool to work with fully static runtimes, I think it would be totally sufficient to just standardize the $TARGET_APPIMAGE feature. All compliant runtimes would need to support it, if it wouldn't work we could make users report that as a bug then. In the same moment, we could explicitly mention that both static and shared builds are permitted.

To communicate that the feature is available, we need to bump the AppImage type/version in some way. In my opinion, it makes no sense to release an entirely new type-3 at this point. What we could do is release a type 2.1 that adds these features. The current magic bytes embedding is not capable of this, though. Therefore, we could pick up the ideas from #26 and add a second(!) more fine grained version marker. This serves two purposes: first, it allows us a "soft start" into actually versioning and releasing the spec, something we have wanted to do for a while already. Second, by adding some "minor" version number (or even going full semver), tools like AppImageLauncher can reliably detect such new features like $TARGET_APPIMAGE. Third, by adding the version marker as a second feature, all tools that currently rely on the type-2 MIME type (or otherwise use the magic bytes) will continue to work. After all, we just add new features, but the type itself is the same and everything will continue to work. Only tools that need new features need to check for the new marker as well. Third, this serves as a test bed for the new proposed type-3 version marker that will replace the magic bytes at offset 0x8 in the long term to improve compatibility with various loaders.

TL;DR: let's add $TARGET_APPIMAGE and a new, secondary AppImage type marker to the specification.

[probonopd]

$TARGET_APPIMAGE could be considered for 1.0; a secondary marker strikes me as overly complicated. It seems to be a valid approximation to assume that the vast majority of type-2 AppImages support it.

[TheAssassin]

You should really read the issue properly, which recaps our in-detail discussion from last weekend. As explained above, your assumption just doesn't hold true. Degrading support is not a sensible option, especially given that this version marker does not break compatibility at all and overall is implemented with ease in just a few lines of linker script.

It is very frustrating having to repeat stuff already explained just above in responses to you.

[probonopd]

Requires discussion.

[TheAssassin]

Some more context: There is no question AppImageLauncher's binfmt_misc workaround sucks. The blog post that explains it in details clearly states all the disadvantages. It has worked extraordinarily well for years nevertheless. To my own surprise, I may add.

Since AppImageLauncher has a very large user base, we cannot just essentially break all newly produced AppImages for an indefinite amount of time. Not only is this very annoying, it's also bad PR for both projects. We should try to avoid that.

The proposal entirely takes out the guesswork: we do not have to make assumptions on whether $TARGET_APPIMAGE is available, we make it a requirement of all compatible runtimes. We can reliably check for that even. Same goes for the new requirement of statically linked runtimes.

This way, one of the two crude hacks required to make AppImageLauncher work can be eliminated, at least for all new AppImages.

[TheAssassin]

I'm just mentioning this for completion. An alternative solution to avoid the use of a version number would be to add some completely new(!) concept of "feature tags", e.g., having a section in the ELF header that contains known-good (ideally versioned, e.g., _1) string markers of optional features. But for this specific case, we need to make the TARGET_APPIMAGE feature mandatory anyway. This should reflect in the version number.

And yes, the AppImage type is a version number essentially. Even if it's just as basic as possible.

I just described an (IMO) better, more advanced version scheme in https://github.com/AppImage/AppImageSpec/issues/39. This version scheme details many open questions that have not been discussed in this issue as of yet. I am confident that we'll find a good version scheme.

[shoogle]

Fine-grained versioning (e.g. type 2.1) and feature tagging (e.g. supports TARGET_APPIMAGE) are welcome additions to the AppImage specification, but it will take time for these features to be implemented in downstream projects, and it won't help with existing AppImages that are already in circulation.

A method that might work with current AppImages would be for AppImageLaucher to register itself as the binfmt_misc handler for the .AppImage file extension instead of using the magic bits. When it's time to run an AppImage, AppImageLaucher can create a soft (or hard?) link to the AppImage at another location, this time omitting the file extension to avoid the infinite loop.

For example, when running /Applications/MyProgram.AppImage, AppImageLauncher would essentially do this:

ln [-s] /Applications/MyProgram.AppImage /tmp/MyProgram  # create link without file extension
exec /tmp/MyProgram                                      # execute the link

I'm not sure whether the kernel would resolve a symbolic link before or after calling the binfmt_misc handler. If it happens before then this method probably wouldn't work with soft links, so you'd have to use a hard link (or mountpoint?) instead. Hard links don't work across filesystems, so using a soft link or mountpoint would be preferred if possible.