ELF resources

[probonopd]

As mentioned in #6 we need a better way to store additional information in an AppImage, outside of its main compressed filesystem image.

Currently we are using ELF sections to store various metadata, but this is not elegant. (While I am writing this I am asking myself: Why is this not elegant? Why do we need something else?)

So we are looking for a way to store additional metadata in AppImages without having to touch the main compressed filesystem image.

In other words, we are looking for a way to store key-values in executables. Here are the existing ones I know of:

• ELFIcon by @glensc, despite its name, is a full-fledged resource system not limited to icons ("resource files should be used in order to store images in a standardized way, this will also allow applications to store other data"). Spec: https://wiki.ubuntu.com/ELFIconSpec, implementation: https://github.com/pld-linux/elficon
• Haiku OS file resources: https://www.haiku-os.org/docs/api/classBResources.html. I have written about them at https://medium.com/@probonopd/my-sixth-day-with-haiku-under-the-hood-of-resources-icons-and-packages-abec8d0e4ec6. There is even a GUI tool to edit those resources: https://github.com/HaikuArchives/Resourcer. If we would use this system, of course we would need to define our own resources, e.g., UPDI (for update information), etc. Maybe @waddlesplash or @pulkomandy have an opinion on how suitable Haiku OS file resources would be in the Linux world?
• FatELF by @rcgordon is a file format that embeds multiple ELF binaries for different architectures into one file: http://icculus.org/fatelf/ Possibly it could be extended to hold generic resources in addition to ELF binaries?

What would be the downsides of using one of those, rather than defining something AppImage specific? By pushing something that is not AppImage specific, chances would be much better to get support into e.g., file managers (so that they can show proper icons without the need for desktop files or thumbnailers)

[TheAssassin]

Currently we are using ELF sections to store various metadata, but this is not elegant. (While I am writing this I am asking myself: Why is this not elegant? Why do we need something else?)

Well, if you store metadata outside the ELF part, you gain many advantages. For instance, the entire runtime could be signed by us and this could be checked easily (i.e., without having to skip sections and other things). When adding metadata, one does not have to fiddle with these weird sections, there's no dependencies on external tools, the complexity is very low compared to what's done now. This has been discussed in other issues already

The "AppImage header" is just about AppImage-specific metadata like update information or signatures. We can investigate alternative specs as well, if they're as portable as our approach. IMO we should however just try to get a type 3 done and could later improve on this in a fourth type.

Haiku and FatELF are bad examples by the way (not sure about ELFicon): both will not run on a regular Linux system, they need kernel-side support. That's a kind of dependency we don't want at all. ELFIcon looks somewhat dead anyway.

[probonopd]

signed by us and this could be checked easily (i.e., without having to skip sections and other things)

Imho whenever you checksum/sign a file and then want to store the checksum/signature inside the file you have checksummed/signed, you will need to exclude the place where you store the checksum/signature from the checksumming/signature checking mechanism. But you should not skip much more than that, or else the whole effort is for nothing. No matter which format you use. Am I wong? (How is this done for deb and rpm files?) I'd be interested how ELFIcon, Haiku resources, and FatELF deal with signatures and/or checksums.

[probonopd]

The "AppImage header" is just about AppImage-specific metadata like update information or signatures

That's my point, we are trying to find a way to store resources, there is noting AppImage specific about this. E.g., an icon stored there has nothing AppImage specific about it. By inventing something that is too closely tied to AppImage, we would guarantee that e.g., RHEL will never automatically show icons stored at that location, because they will never do work that is "only" about AppImages. This is why I am looking for a more generic solution.

[probonopd]

need kernel-side support

Well, for FatELF the objective is to run ELFs stored inside those "fat" files, and I can see how the low-level OS (kernel or at least linker/loader) needs to support this. For non-executable resources like icons and update information I don't see why you'd need special kernel support. Not sure about why Haiku resources would need kernel support.

[TheAssassin]

That's my point, we are trying to find a way to store resources, there is noting AppImage specific about this.

I'm talking about metadata, not resources. They're different things. That's what I want to point out.

For non-executable resources like icons and update information I don't see why you'd need special kernel support.

If they're real specifications which modify the ELF format and don't hack that in there (like AppImage does right now), they will need some special system support. Even a modified linker is a no-go, as it breaks portability.

[probonopd]

I'm talking about metadata, not resources. They're different things. That's what I want to point out.

To me, metadata is a special kind of resources.

E.g., checksums, signatures, icons, translated strings,... basically everything (and more) that today is referenced in an AppStream metainfo file and in a desktop file.

[pulkomandy]

Haiku ELF resources are simply appended at the end of the file. Since ELF section header specifies a size, that extra data is simply ignored by Haiku's elf loader.

This is a bit annoying to manage, for example using strip or any other binutils based tool (objcopy, ...) will remove the resources.

We will probably change this in Haiku at some point to use a dedicated ELF section instead. That seems more elegant to me.

[probonopd]

Thanks @pulkomandy for chiming in here. So do you think the Haiku resource format could work on Linux?

Haiku ELF resources are simply appended at the end of the file

That sounds a lot like what @TheAssassin is proposing here:

The new AppImage file format might therefore look like this:

+------------------+
| AppImage runtime |
|   (ELF binary)   |
+------------------+
| AppImage header  |  <-- new
+------------------+
|                  |
|     payload      |
| (squashfs image) |
|                  |
.                  .
.                  .
.                  .
+------------------+

[TheAssassin]

Well, nobody'll run strip on an AppImage. This approach is used by many projects anyway. Binary packers, self-extracting archives, ...

[probonopd]

Indeed I had been thinking of putting the payload filesystem image into an ELF section of its own, which would make it a "really standard" ELF. Why not? As long as there is (or we can make) a tool to add the ELF section for an existing ELF. It would feel a lot "cleaner" to me. How do you normally say, what we now have is "a hack"?

Only reason I didn't do it is that I don't understand enough about ELF to have a good sense of pros and cons.

[icculus]

Just popping in to reaffirm this is not a good fit for FatELF, as others pointed out.

My suspicion is that the technically correct (but not easiest) solution is to define a new ELF section. But the path of least resistance here is Haiku's approach.

--ryan.

[mgord9518]

Mentioned here, I think using a ZIP appended to the end of an ELF would be a good way to store resources inside an executable.

It can be made very generic as to not be specific to AppImage, portable and incredibly easy to implement in any language as the file could simply be opened as a zip. Icons and other resource names would have to be standardized. An example could be the zip root with a directory "icons", under it being "16.png", "48.png", "64.png" etc. Keeping all file names static eg: "desktop_entry" instead of "[APP NAME].desktop" would also simplify extraction of resources

[icculus]

I haven't read this whole discussion to see if someone mentioned this, but please don't append a zip to an ELF as a standard, because then you remove the ability for anyone else to append a zip to an ELF (self-extracting archives, games built into a single binary, etc).

ELF files can have arbitrary records that existing ELF loaders can ignore, just add a new one...?

[mgord9518]

It wouldn't remove the ability as either they could share the same zip file (store resources under a subdirectory instead of the root), or simply not use it as it would be completely optional eye candy

I think the odds of any solution here being used outside of AppImage is incredibly unlikely as there are already existing solutions that aren't being standardized. The zip route is interesting to me as it doesn't just stop at ELF files, it could potentially be a way to add resources to bytecode, scripts, etc.