embed all relevant metadata as ELF sections

[lawl]

I'm personally unhappy with the currently available AppImage tooling (especially package-manager-ish). I was considering writing my own tooling, and then realized I need to pull things like the icon out of the sqashfs/ISO 9660.

Is there a specific reason to not have the icon and desktop entry embedded as ELF sections so that tooling that deals with AppImages can easily access them without having to support multiple archive formats (squashfs/ISO 9660)?

It seems insane to me that a tool that e.g. creates thumbnails in a file manager has to parse an ELF and then have a squashfs driver just to pull out a tiny PNG file.

[antony-jr]

@lawl you can use https://github.com/AppImage/libappimage to get the icon and everything you need.

Putting icons and desktop file in ELF section is totally not recommend and does not confirm to the AppDir specification.

EDIT: libappimage also has a easy function call to integrate a AppImage to the system.

[lawl]

Thanks, but I didn't ask for help using a library, I created an issue on the spec because the spec seems insane to me and I wanted to ask what/if the specific reason for that is. And depending on that the potential answer would of course be if a potential Type 3 AppImage could change that.

[antony-jr]

As I said earlier there are no plans for doing this. Especially not the desktop file and icons. There is a reason to it. Icons are not simple as you think. Icon can have different sizes and how do you propose to place different sizes of icons in the ELF section headers, even if you do that, it won't be neat at all. Also AppDir Specification does not allow this https://docs.appimage.org/reference/appdir.html

TL;DR: Not a very good idea, just use libappimage and continue with your project.

IMO: The AppDir format should not be changed in Type 3, the only thing that needs to change is the signature problem and musl support.

[lawl]

I see, the spec must not be changed because the spec does not already allow it! Makes sense, thanks for your help.

[antony-jr]

I see, the spec must not be changed because the spec does not already allow it! Makes sense, thanks for your help.

I meant the ROX Desktop one which this AppDir inherited. (but that is not enforced anyways.)

[TheAssassin]

@antony-jr as far as I can see, this is more about a type 3 rather than introducing breaking changes for the existing type 2.

I don't know where @antony-jr's information is coming from. I am not aware of discussions let alone decisions on most of these topics. So far, only some basic drafting has been done on a type 3. I'll try to clarify a few points from my point of view.

The main reason this is a bad idea is that at this point, compatibility to the AppDir standard, which was defined externally and only extended by AppImage, is still required. It makes no sense to store those data redundantly. Plus, ELF sections are annoying to maintain (and parse). I would like to get rid of these entirely. This would, in the future, allow signing the entire AppImage runtime, and validating such signatures. Right now, signature checking on the runtime is relatively complex, as sections in the middle of a file have to be skipped. That's nontrivial to implement. (I wrote the validation code in AppImageUpdate and in the process had to implement quite a bunch of fixes and workarounds in the runtime and other places to actually make signatures useful at all; for instance, keys were not bundled before, but without a public key to check it with, a signature is just bloat.)

A future AppImage type 3 (https://github.com/TheAssassin/type3-runtime, WIP) specification will most likely abandon some of the ROX filer AppDir specifications. The desktop entry and the icon in the AppDir root are not required during runtime. As you correctly recognized, the main use for the icon is thumbnailing. Thumbnails are typically generated from a single image, not multiple ones like with desktop integration. This thumbnail source image however does not really have to be located in the root of the payload. However, I don't think ELF sections are a great fit either. In my type 3 draft, I invented an easier-to-parse "AppImage header" structure located in between ELF runtime and payload(s). This header implements strict versioning (which will already help with many implementation related issues) with strong backwards compatibility.

Talking about redundancy, the desktop entry in the AppDir root is usually redundant to a second entry in usr/share/applications. This is not a huge problem, as squashfs, the payload format used for type 2, is good at deduplication, and the entry can also be a symlink. However, to an integration algorithm, it might be unclear which one to use. They are not guaranteed to have the same contents by the specification. Therefore, tools like linuxdeploy just symlink in either direction.

I am a strong advocate for using an FHS-inspired directory layout within the AppDir, which is a widely known and accepted layout and also supported by many build tools as an output layout (including CMake, meson, autotools, qmake, and many more). Also, I have been advocating for creating a proper AppDir specification and maintaining it independently from the AppImage specification. linuxdeploy is an AppDir maintenance, not an AppImage creation tool. AppImage is only one possible output format (and, honestly, currently the only one).

TL;DR: A type 3 should only have one file in the root: AppRun. Thumbnails should in the future be moved into a section in the "AppImage header". The AppDir should strictly follow the FHS. A real AppDir spec should finally standardize the layout within payloads. The desktop integration files should be in AppDir/usr/{applications,icons}. Note that for desktop integration, you do not necessarily need to be able to parse the payload format. The runtime already allows the extraction of files and also mounting AppImages. A type 3 would just have to specify that runtimes have to implement both a way to list all files and a way to extract single files. But I'd like to even specify possible payload formats.

It is not a big problem that type 2 is a bit messy in some regards, as libappimage contains a lot of compatibility code to support all kinds of existing deployment schemes. However, maintaining libappimage is labor-intensive. I don't like hunting bugs there. The amount of code is suboptimal, too. When working on a future type 3, we should try to properly specify and standardize all kinds of aspects. Standardization is not the enemy, it's the solution to a vast amount of issues with AppImage.

As I got fed up with the climate in the AppImage project, I don't work on a future type 3 within the AppImage organization any more. Instead, I started the type 3 runtime project in my account and later founded https://github.com/AppImageX in order to engineer the future of AppImage. This allows me to do things from an engineer's point of view rather than trying to maintain bad ideas from the past by all means. Please feel free to comment on my type 3 draft. But please note that I have not got to working a lot on this type 3 a lot in the past year and it won't get better in the next months. The project has not been abandoned, though. It's just time intensive to design something from the ground up, making security the first and foremost goal.

@antony-jr if you are interested in developing an all-new, standards-driven AppImage, please don't hesitate to get in touch either. But that means that some quirks of the existing mess have to go. While full backwards compatibility is always nice to have, it's still just nice to have.

Let's first engineer a proper concept, then implement a solution, not vice versa.

Edit: I didn't mean to say "type 2 is basically broken beyond repair", but the comment ended up like that. Please don't be offended. I'm speaking from an engineer's point of view. There's things that are broken much worse, e.g., the entire VoIP protocol mess (SIP, SDP, RTP, ...).

[antony-jr]

I don't know where @antony-jr's information is coming from. I am not aware of discussions let alone decisions on most of these topics. So far, only some basic drafting has been done on a type 3.

I just guessed. lol. :)

Plus, ELF sections are annoying to maintain (and parse). I would like to get rid of these entirely. This would, in the future, allow signing the entire AppImage runtime, and validating such signatures. Right now, signature checking on the runtime is relatively complex, as sections in the middle of a file have to be skipped. That's nontrivial to implement. (I wrote the validation code in AppImageUpdate and in the process had to implement quite a bunch of fixes and workarounds in the runtime and other places to actually make signatures useful at all; for instance, keys were not bundled before, but without a public key to check it with, a signature is just bloat.)

I totally agree with this. The reason why I never touched the signature verification, it's just messy with ELF headers.

I invented an easier-to-parse "AppImage header" structure located in between ELF runtime and payload(s). This header implements strict versioning (which will already help with many implementation related issues) with strong backwards compatibility.

Neat!

However, maintaining libappimage is labor-intensive. I don't like hunting bugs there. The amount of code is suboptimal, too. When working on a future type 3, we should try to properly specify and standardize all kinds of aspects.

I get you @TheAssassin , you seem to be the sole maintainer of all core things AppImage.

As I got fed up with the climate in the AppImage project, I don't work on a future type 3 within the AppImage organization any more. Instead, I started the type 3 runtime project in my account and later founded https://github.com/AppImageX in order to engineer the future of AppImage.

Interesting proposal, I see that you intend to use rust instead of C. I just got into rust and I'm using it a lot.

@antony-jr if you are interested in developing an all-new, standards-driven AppImage, please don't hesitate to get in touch either. But that means that some quirks of the existing mess have to go. While full backwards compatibility is always nice to have, it's still just nice to have.

:+1:

[lawl]

@TheAssassin thanks for taking the time to reply, that was way more indepth than i could have ever wished for :)

You are probably right that using a section header per metadata unit is a bad idea. I really meant that these things really shouldn't (only) live inside the sqaushfs. Though I'm not really sure duplication is such a big deal.

As it stands right now, AppImage offers a good experience if you just want to try out an application, but I think there's a lot of potential still in innovating on decentralized package management. But for it to be a package format that just so happens to also be a runnable ELF, I think having metadata readily accessible is vital.

There's some other things I agree/disagree with, or simply personally don't care about, but I'll go check out your type 3 project and maybe leave more detailed feedback there.

As it stands today, distributing apps to desktop linux from my perspective is still an absolute nightmare, and I'm happy to talk to anyone interested in fixing it.

[TheAssassin]

I get you @TheAssassin , you seem to be the sole maintainer of all core things AppImage.

De facto most of the core work in the last years has been done by me. However, that caused some burn out effect with regards to AppImage. Also, AppImage has no real governance model, and discussions on have never led to any progress or change. Thus, I continue to maintain everything as-is, but there are no more future development efforts from my side within this GitHub organization. I hope one day a small team of engineers will form some kind of organization to give AppImage a stable and sustainable future.

Interesting proposal, I see that you intend to use rust instead of C. I just got into rust and I'm using it a lot.

One thing Rust is not good at is working with C libraries. The only viable way seems to be FFI, but that can be pretty annoying. We do want to continue using squashfuse, otherwise we have to rewrite it in Rust, but that goes against keeping the amount of code we have to manage small. I'm not sure whether we should just go for C++ therefore, at least for the runtime. It would make a lot of things significantly easier and safer (RAII for the win). What I like about Rust is its built-in code safety features. And if there is one thing that should be as rock solid as possible, it's the AppImage runtime. It runs in thousands of AppImages on tens of thousands of computers every day.

For non-runtime components (i.e., tooling), @srevinsaju and I have been playing with the idea of just going for Python. Bundling Python tools into AppImages is relatively easy nowadays, and demonstrated in many tools I wrote, most recently https://github.com/TheAssassin/pyuploadtool/. C++ is a great language, but it can be overwhelming. And Rust is still a niche language, although I'm impressed that there are more and more complex tools, e.g., ones with Gtk+ UIs like Popsicle.

As it stands right now, AppImage offers a good experience if you just want to try out an application, but I think there's a lot of potential still in innovating on decentralized package management. But for it to be a package format that just so happens to also be a runnable ELF, I think having metadata readily accessible is vital.

We should probably talk about the kind of metadata that is really vital to tools working with AppImages over at the other repo.

By the way, I just thought, why not bundle (desktop integration) metadata separately? I think we could probably just zip them and append them to the main payload. Of course, that means that some of it will be bundled redundantly, but it's just a few kB of icons and desktop entries. It adds complexity, though, and requires excellent tooling to keep the overhead as small as possible.

Talking about overhead, even today, keeping it as low as possible is important. Bandwidth can be expensive for users, and also from a green IT point of view, every byte that needs to be transferred costs energy. That's why AppImageUpdate greatly takes advantage of having squashfs instead of, e.g., zip archives, as a payload type. In squashfs images, files are block-aligned, meaning that the same file contents create the same blocks. Thus, if a file does not change, it doesn't have to be retransmitted.

[TheAssassin]

By the way, ever since I have been using my tool AppImageLauncher, I have had no issues with using AppImages like any other system applications. libappimage works fine for way over 95% of all AppImages (there's always some broken ones). Even MIME type integration is working flawlessly. All this makes AppImages a first-class citizen on the Linux desktop.

I am aware that AppImageLauncher might not implement everyone's favorite workflows, but that's the great advantage of having a library that implements the algorithms and can be used by any kind of integration tool. And once we properly standardize those algorithms so that people can reimplement them without having to whitebox reverse engineer libappimage (we have done a lot of this before creating a dedicated libappimage), I can imagine a whole bunch of new tools to be developed. (Though, I think AppImageLauncher is still a good fit for all everyday average Linux users.)

[antony-jr]

For non-runtime components (i.e., tooling), @srevinsaju and I have been playing with the idea of just going for Python. Bundling Python tools into AppImages is relatively easy nowadays, and demonstrated in many tools I wrote, most recently https://github.com/TheAssassin/pyuploadtool/

I would still recommend using rust because it avoids runtime bugs that get noticed only on production. Also the binary size is really huge, If we go with rust we can static compile it to less than 5 MiB.

But I really don't want to argue on what language to use but IMO python should be used as glue language.

[srevinsaju]

I would like to see a future appimagetool written in python (as it's rich in libraries, and highly extensible), but the runtime could be preferably be written in Rust, as suggested.

[TheAssassin]

5 MiB is pretty large for a runtime, actually. The existing runtime is way under 1 MiB. Remember that every AppImage has to ship this header, and it adds up quickly. A hundred AppImages shouldn't occupy half a GiB of runtimes.

I would like to see a future appimagetool written in python (as it's rich in libraries, and highly extensible), but the runtime could be preferably be written in Rust, as suggested.

Exactly that's what I was trying to suggest. Obviously, a Python runtime would not work.

[antony-jr]

5 MiB is pretty large for a runtime, actually. The existing runtime is way under 1 MiB. Remember that every AppImage has to ship this header, and it adds up quickly. A hundred AppImages shouldn't occupy half a GiB of runtimes.

Sorry to mislead, I was actually talking about the non-runtime tools components. Like the appimagetool.

EDIT: There is already a go implementation https://github.com/probonopd/go-appimage which seems to be much more portable, faster and reliable than python. But still rust would beat go in regards to speed.

[lawl]

Alright since this has gone completely off topic, I take the answer to that is a "no" and will close this issue before some starts debating editors.

[probonopd]

Hi @lawl since I am the one to blame for this I will give my answer:

It seems insane to me that a tool that e.g. creates thumbnails in a file manager has to parse an ELF and then have a squashfs driver just to pull out a tiny PNG file.

Legitimate question. So let me explain:

An AppImage is a filesystem image ("disk image") that gets mounted when you double-click (execute) it. That is its core way of working.

Unfortunately, "Linux" (really: the userland) has no real standard of embedding icons and other metadata into executables (ELF files), otherwise we would be using that. The way icons work in "Linux" is that they are separate files, separate from the executable ELF file. (I find this VERY stupid and annoying myself but https://github.com/pld-linux/elficon never was merged into the Linux kernel.)

So we just put those icon files, like all other files, into the filesystem image.

Now, to extract the filesystem image, you need to know at which offset in the AppImage file the filesystem image starts. One way to find this offset out is by calculating the size of the ELF file that is prepended to the filesystem image inside the AppImage file.

I'm personally unhappy with the currently available AppImage tooling (especially package-manager-ish). I was considering writing my own tooling, and then realized I need to pull things like the icon out of the sqashfs/ISO 9660.

I agree that "Linux" (read: the userland) needs a way of storing and accessing resources inside files.

Haiku does it better: https://medium.com/@probonopd/my-sixth-day-with-haiku-under-the-hood-of-resources-icons-and-packages-abec8d0e4ec6

Is there a specific reason to not have the icon and desktop entry embedded as ELF sections so that tooling that deals with AppImages can easily access them without having to support multiple archive formats (squashfs/ISO 9660)?

When we made the spec, we wanted to keep the AppImage format as simple as possible, wanted to keep everything compressed, and avoid redundant information.

But I tend to agree that a generic resource system (think Mac resources, Haiku resources) implemented using one or more ELF sections is a thing that we shoud look into, especially since we have started going down this path for things like update information and signatures anyhow.

Someone should make, and standardize, a generic system to have a "key-value store" in ELFs that could hold arbitrary information, inluding icons. And ideally it should not be specific to AppImages. Then we might consider it in a future version of the AppImageSpec (although we want to change that spec as infrequently as possible).