Clarify Image format, runtime, header

[probonopd]

Hi @TheAssassin. In order to get a serious discussion started around the ideas and concepts in this repository, we should clarify the use of some terms first, so that everyone (including the two of us) understands what is meant. Reading in this repository, I think some terms are mixed and matched in ways that make them less clear to understand.

So here is my proposal:

• Image format. Describes the file format of a spec-compliant AppImage. (Originally I was tempted to call this the "AppImage version", but that term could mean too many things, so I went with "image format".) The Image format can be type 0, type 1, or type 2. Are you proposing to introduce a type 3 image format? Then let's draft a section for it that would go into https://github.com/AppImage/AppImageSpec/blob/master/draft.md#image-format . I understand that one of the killer features of a type 3 image format would be that the magic bytes would at another offset. Which offset are you proposing? The image format is part of the AppImageSpec which is under the MIT License
• Runtime. An AppImage runtime is the executable code that is placed at the beginning of an AppImage file. Each runtime usually implements the mounting and running of AppImages that are in a certain image format. So if we define a type 3 image format, then we also need a runtime that can deal with this image format. Nothing forbids a runtime to support more than one image format, by the way. So if the only change between type 2 and type 3 is the location of the magic bytes, then the executable code of the runtime might otherwise well be the same (so that the same runtime code could run both). The current runtime is under the MIT License
• Resources. The term "AppImage header" you are using is confusing me. As you know, at the beginning ("head") of each AppImage file there is an ELF. The ELF has a "header" that is defined in the ELF standard. Are you looking for a better way to store additional information in an AppImage, outside of its main compressed filesystem image? Currently we are using ELF sections to store various metadata, but I'd agree that this approach is less than elegant. What would be needed is a more flexible way to store additional metadata in AppImages without having to touch the main compressed filesystem image. Instead of inventing yet-another AppImage-specific clutch, however, I would hope that we can add/find a way to achieve this with ELF. I would propose to call this "(ELF) resources" rather than "header" by the way. Let's think of resources as key-value pairs that can be added to ELFs (and hence, AppImages). Haiku OS seems to have a way of adding metadata to ELF files. Why can't we use their mechanism, and specify our own metadata keys?

[probonopd]

Imho, we should decouple the discussion of the three things from each other.

First, we should define the type 3 image format. This may require us to agree on a way to deal with resources first (I would love to see a non AppImage-specific way to deal with resources - after all they can be useful for non-AppImages too)

Second we will then have to adjust the existing runtime (or write an entirely new one) to support this new image format

[TheAssassin]

Right now, the repository is a mix of type 3 specification and runtime reference implementation. In the future, we can formalize the spec and move it over to any repository (although I'm pretty sure the MIT license is not a good idea).

Until the type 3 spec hasn't been fully engineered here, I'd rather not start to do things in the spec repo. The reason is that this spec repo lacks formalization, e.g., there's no versioning of any kind, and it calls itself a draft even after all the years. A type 3 should have more reliability for third party implementations. For instance, we're already working on things like compatibility schemes which should ensure long-time stability.

The change of the magic bytes' location is not the only change with regards to the image layout. You might've noticed that I plan on including an "AppImage header" carrying all the metainformation currently scattered across the ELF header (and I think we should rather not tinker with that too much) and the contents of the AppImage. The AppImage header will be able to carry any kind of information, is fully specified and backwards compatible as well as versioned. I came up with a model how we can add new features without breaking software that only supports AppImage headers up to version X. Also, we can store arbitrarily sized information in it, i.e., we don't have to reserve huge buffers and hope for the data to fit in there.

Introducing a blob in between the regular ELF runtime header and the payload has several advantages. You can find most of them across the various issues. Most notably:

1) we do not have to modify the ELF runtime at all when composing AppImages 2) this allows for signing the entire ELF runtime from AppImage's side (which could be used to, say, show a hint "uses an official AppImage runtime) without any complexity (such as e.g., skipping sections when creating/verifying signatures) 3) you just need to calculate the size of the ELF runtime to find the AppImage header, then you can just read that from there (instead of having to understand the ELF section table and reading multiple sections) 4) it doesn't depend on buffer sizes (right now, if you wanted to support different sizes for signatures, you'd have to rewrite the entire ELF file; hence, type 2 doesn't do it, but just reserves multiples of 1 kiB for every AppImage-related section)

We could even discuss moving signatures of the AppImage to the end of the AppImage, maybe even in ASCII armored form? That way, it might be possible to check signatures of AppImages without any additional complexity, as gpg2 might just support that? Someone should really try that.

[probonopd]

"AppImage header"

We should discuss this idea in more depth. Currently I have a tendency not to be in favor of it (because it complicates things significantly) but maybe I am not understanding it fully yet. Maybe we should set up a date and time for an Jitsi session?