Why another format?

[probonopd]

Hi @xplshn, I'd be interested in what pelf wants to do differently than AppImage, and whether you'd think that the same could be achieved by modifying the existing AppImage format. Is it really necessary to have yet-another-similar-format?

Thanks for your consideration.

[xplshn]

Hi @probonopd. The idea here is to provide a simpler executable format. With no-recompilation needed. What PELF does is it collects all of the dependencies of a program into a single file automatically, the resulting file is an .AppBundle, which will set LD_LIBRARY_PATH and other variables so that you programs works as if its dependencies were installed. .AppBundles avoid calling the outside libraries as much as possible.

Another difference is that AppBundles' loaders will run everywhere, so, you can have this work under FreeBSD, OpenBSD, Alpine Linux (busybox), without having to install any other programs, since its just (POSIX) shell.

PELF only exists because AppImages are unhackable. They require external tools in order to be created, they require a runtime which is compiled, they don't contain their dependencies inside, etc. Also, AppImages cannot be modified with a text editor once they have been created, it is often handy to change parts of the loader.

I don't intend AppBundles to replace AppImages. The idea is not to compile programs again, but to distribute already compiled programs with minimal effort, you just need to write a small shell wrapper some times to package things like Chromium, but that's all it requires.

And these are based in the concept of "sets", for example, my Window Manager is not installed, its just an .AppBundle which contains all of the X programs I use inside of it, along with their dependencies. Programs running inside of my WM and my WM itself are able to execute these files regardless, same for my browser, it has xdg-open inside, but I don't really have xdg-open installed...

[xplshn]

If the programs and libraries you package with PELF have been compiled with their LibC statically linked, even if they reference other .so files, they should work on any ABI-compatible OS, since the LibC bits are static. I suppose this applies to AppImages too, but I'm yet to see AppImages that run on Musl without having to be re-compiled specifically for it.

Another approach would be adding Musl itself to the bundle, and using Patchelf(from NIX) to link it on-the-fly, so that truly independent bundles can be generated and ran on Glibc/Musl. I've tried it before, but it meant having the TMPDIRS hardcoded so that the binaries find the re-linked libc.

[probonopd]

Coudn't most if not all your describe be done with AppImage as well? I mean, what you put inside an AppImage is entirely up to you.

[xplshn]

Coudn't most if not all your describe be done with AppImage as well? I mean, what you put inside an AppImage is entirely up to you.

I guess. The major problem is AppImages not being able to run under Musl (which is what I use). That and the fact that it requires lots of external programs to be created + crafting an AppDir. This way easier & simpler, and it is the same for FreeBSD, OpenBSD, Void, Alpine, Debian and any other platform with a degree of POSIX compliance: pelf /usr/bin/spectrwm ~/.local/bin/spectrwm.AppBundle --add-binary /usr/bin/scrot --add-binary /usr/bin/wezterm --add-binary /usr/bin/wezterm-gui --add-binary /usr/bin/rofi

AppImages, AFAIK, only cover Glibc-Linux and HelloSystem.

[probonopd]

The major problem is AppImages not being able to run under Musl (which is what I use).

I think this is no longer true with https://github.com/AppImage/type2-runtime/releases

If you cannot run AppImages created with it under Musl, then it is a bug (though I am pretty sure that it works).

the fact that it requires lots of external programs to be created + crafting an AppDir

That's simply a matter of the tools that create what goes into an AppImage. If you have tools that are easier, why not let them create an AppImage?

[xplshn]

I think this is no longer true with https://github.com/AppImage/type2-runtime/releases

I wasn't aware of that. Nice! Does it work under NetBSD & OpenBSD too?

That's simply a matter of the tools that create what goes into an AppImage. If you have tools that are easier, why not let them create an AppImage?

I guess that could be a future project

[probonopd]

Only tested on FreeBSD and there it works.

[probonopd]

Does pelf use FUSE?

[xplshn]

Does pelf use FUSE?

Nope. It only relies on tar, but that can be changed to any other archiver too.

[xplshn]

Can I consider this issue closed? If you have further questions I'll answer them.

[probonopd]

Yes, the original question is answered. It's great to see someone interested in promoting the state of bundle formats. I still have some questions, so thanks for answering them.

Nope. It only relies on tar, but that can be changed to any other archiver too.

What happens if you have, e.g., a 3 GB pelf (a game, for example). Will the whole contents be extracted to the filesystem before the payload can be executed (which would be extremely bad for startup times and disk space consumption), or is it extracting things on-the-fly as needed?

[xplshn]

Yes, the original question is answered. It's great to see someone interested in promoting the state of bundle formats. I still have some questions, so thanks for answering them.

Nope. It only relies on tar, but that can be changed to any other archiver too.

What happens if you have, e.g., a 3 GB pelf (a game, for example). Will the whole contents be extracted to the filesystem before the payload can be executed (which would be extremely bad for startup times and disk space consumption), or is it extracting things on-the-fly as needed?

I'm glad you brought that up. PELF works in a naive way, it extracts to your TMP. I'm still wondering how I could change that. Since this is shell, it has to be a command with few dependencies, and if possible one that's available in SUSV4-compliant systems.

is it extracting things on-the-fly as needed?

How do .AppImages do this? Would it work in other systems apart from Linux and those with a compatibility layer for it like FreeBSD?

[probonopd]

So if your compressed PELF is 3 GB, then it extracts e.g, 6-9 GB before it even starts the application? Now, that'd be a showstopper for me. It takes too long and more importantly, if you are running from a Live ISO on a 4 GB RAM machine, it won't even work.

AppImage works by mounting rather extracting files. AppImages are self-mounting disk images. The mounting is almost instant. Nothing gets extracted to disk. No disk space is needed.

[xplshn]

So if your compressed PELF is 3 GB, then it extracts e.g, 6-9 GB before it even starts the application? Now, that'd be a showstopper for me.

AppImage works by mounting rather extracting files. AppImages are self-mounting disk images. Nothing gets extracted to disk.

I understand that. I just haven't figured out a way yet to do something like that and have bundles work across systems. For small programs, it should be fine, even for Chromium, the binary ends up taking about 270~ megs, and start times are fast even on my +8 years old notebook with no graphics and an i3 of 7th gen.

I believe it could be implemented using Cosmopolitan lib C and then including that in the payload/loader

[xplshn]

This is what I envision: If your system has FUSE and the tools needed to mount -> Mount the archive with the contents of the bundle Else -> Extract to /tmp as usual

I'll probably have to write an optional runtime. SH will be the primary runtime that loads the second runtime, which would do work a lot more involved. This is much better than meddling with ELF and magic numbers in my opinion.

[probonopd]

You need ELF so that double clicking in the file manager works reliably. (Early predecessors to AppImage used shell scripts, and it never worked reliably with all file managers.)

[probonopd]

Do you know that ./Some.AppImage --appimage-extract-and-run already exists?

[xplshn]

You need ELF so that double clicking in the file manager works reliably. (Early predecessors to AppImage used shell scripts, and it never worked reliably with all file managers.)

That's why I'm currently working towards an APE-like approach, where the files are recognized as ELFs even though they are a shell script¹. So,

Current roadmap:

1. Versioning.
2. Optionally provide the option to mount the TAR archive instead of instead of copying the files.
3. Employ the same tricks that the APE loader uses to be recognized as an ELF, preferably, implement a tool that can be used to turn any SH script into a (fake) "ELF".
4. Simplify everything by splitting the loader into a very barebones loader and some helper binaries written in shell, for example, the embedded thumbnail generator could be one such helper. The idea being that even if the user isn't able to run the .AppBundle, he can always extract it and repackage it again without having to start from scratch.

5. Consider using the AppDir layout, since it is well-thought out and .AppImages already work this way, thus allowing us to contribute. If only AppImages had their loader run in multiple systems without the need for re-compilation, I think that'd be THE perfect distribution format.

   Current setbacks:

   • Code has to be readable to stay hackable, given that this is SH, it may end up being an unbearably disgusting mess, so I have to be specially careful.
   • Helpers to pack QT and other programs made with intricate toolkits probably won't be supported because of the lack of manpower, and the inability to piggyback from AppDirTool.go since the code is unreadable, at least for me, it is too deeply tied to the Appimage ecosystem. I'd be great if it were independent enough to be compiled/installed with just go install.
   • My time to work on PELF is limited, and there are no other projects like this one apart from AppImages, and I can't benefit much from that ecosystem

Read these two if you are interested in how APE works:

• https://github.com/jart/cosmopolitan/blob/master/ape/loader.c
• https://github.com/jart/cosmopolitan/blob/master/ape/ape.S
  1. https://justine.lol/ape.html

I'm sorry if this too long or unrelated