Open probonopd opened 10 months ago
+1. @hollowaykeanho, I really think this ecosystem should be supported. About the technical deficiencies, you can implement it the way Cargo & PyPi does: make it entirely optional and backpack it on-demand.
There is no doubt AppImage will damage the already statically compiled binary by its packaging system (tracked under: https://github.com/AppImage/AppImageKit/issues/877) but it's still a use case to consider (e.g. one does not need to install flathub just to use flatpak).
Since the AppImage packager is an independent service provider, as long we address the problems out clearly in the documentation, our job should be considered completed.
There is no doubt AppImage will damage the already statically compiled binary by its packaging system
By now, we have the statically linked runtime so that using AppImage no longer makes the system need glibc. When using it to package a static binary, the only requirement on the target system is FUSE and a recent kernel.
the only requirement on the target system is FUSE and a recent kernel
Correct me if I'm wrong. This means for a static binary product (already not depending on anything and even operable in the absent of sysfs like Docker stretch image), after AppImage packaging, the it will now have to depend on sysfs with FUSE.
That's his first point, right?
Will there be a development progress where the packaged AppImage does not depends on external library (as in fuse)?
after AppImage packaging, the it will now have to depend on sysfs with FUSE
I have never tried (in fact, never seen) a system without sysfs, so it would be worth a try.
Will there be a development progress where the packaged AppImage does not depends on external library (as in fuse)?
Yes, by using the static runtime, libfuse is no longer required (but the fusermount
binary is).
I have never tried (in fact, never seen) a system without sysfs, so it would be worth a try.
We successfully implemented with some technologies (notably with Go).
libfuse is no longer required (but the fusermount binary is).
As in, is it possible where AppImage will be shipped without any external dependencies (which means fusermount
included) in some way? It will be bloated yes but it honors the "one file" paradigm properly.
AppImage will be shipped without any external dependencies (which means fusermount included) in some way?
Hi @probonopd, appreciate your responses.
This query is very important because AppImage packager is altering the customer product (for musl and statically compiled case). It's the same case .rpm
packaging system illegally stripping binary by default but we managed to disable it.
A roadmap of such development is also acceptable.
Also, regarding cross-compilations, can AppImage runs on non-amd64 linux system (e.g. aarch64
) or is it a strictly amd64
architecture?
Supporting these 2 pipelines:
filename-universal.appimage
)filename-amd64.appimage
, filename-arm64.appimage
)Supporting these 2 pipelines:
I believe (1) is no and (2) is yes, if this page is valid: https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages/cc2441518975caca23e9ce2dba6f08a22c678d1e#processor-architectures
Great. I guess the packager will behave similarly to Red Hat packagers (rpm & flatpak) since no windows-based or macOS-based version is available (flow: windows X-compile to Linux & darwin X-compile to Linux). Closest packager I can find is: https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages/cc2441518975caca23e9ce2dba6f08a22c678d1e#processor-architectures
Another problem would be sourcing the packager securely. At the moment, no shasum is available and it is not appearing in Homebrew. The easiest fix would be an installer polygot script like https://sh.rustup.rs I guess.
what about the Go version? https://github.com/probonopd/go-appimage/releases/tag/continuous
Will need @probonopd to clarify. All the Go binaries are wrapped inside an AppImage while AppImage is a Linux only thing since it needs fuse library.
This led to my previous forum question since Go is the only one that can cross-compile statically for linux
, darwin
, and windows
platforms with the same source codes without many hassles dealing with the native libraries. The choice of choosing Go as the language for a packager software is definitely correct but wrapping a compiled Go static binary inside an AppImage container seriously ham-struck its own packaging capability (in terms of facility coverage).
The good news is AppImage is still a good container for packaging multiple files product (e.g. e-books web server) but definitely not for those single-app ELF static binary.
Does that means will you support the ecosystem?
Not yet - not until the supply chain is securely stable.
I can't simply script the packaging algorithm like how I did for deb, ipk, DOTNET (nupkg), and Chocolately (nupkg) either since AppImage's low-level is quite complicated.
Let me start by pointing out that AppImage is a self-mounting filesystem image that executes whatever the author of that particular AppImage has put inside.
Technically an AppImage consists of a tiny ELF executable (the "runtime") and a compressed filesystem image (e.g., squashfs with zstandard compression). When one executes an AppImage, the runtime uses FUSE to mount the filesystem, runs the file ./AppRun
inside it (which starts the payload application), and unmounts the filesystem again once the payload application is no longer running.
Traditionally, the runtime linked dynamically to libfuse2 to do this. But then, some (but not all) distributions started shipping libfuse3 instead of libfuse2, with no transition time shipping both. So, with the new static runtime, we don't link dynamically link libfuse anymore. However, the mounting still requires a functional FUSE setup, wich (at least) means that the kernel need to provide /dev/fuse
and there must be a functional fusermount
binary on the system. FUSE must be provided by the system, we cannot bundle FUSE inside the AppImage, because it is partly kernel-provided (e.g, /dev/fuse
) and is partly setuid root (e.g., fusermount
).
I never investigated what "a working FUSE setup" requires "under the hood", as this part is assumed to be managed by the operating system (and indeed, all major mainstream desktop distributions seem to be shipping "a working FUSE setup"). Does it need sysfs? I haven't tested so far.
As in, is it possible where AppImage will be shipped without any external dependencies (which means fusermount included) in some way? It will be bloated yes but it honors the "one file" paradigm properly.
A recent enough kernel and a working FUSE setup are system requirements for AppImage.
Also, regarding cross-compilations, can AppImage runs on non-amd64 linux system (e.g. aarch64) or is it a strictly amd64 architecture?
AppImages are platform-specific because the runtime is a platform-specific ELF. So you can make one AppImage for each architecture, but not one AppImage that runs on all architectures. (For that to work, the AppImage runtime would have to be an interpreted script rather than an ELF executable, which would require an interpreter to be available on the system, and would also have other downsides - been there, done that.)
Multiple appimage per platform (e.g. filename-amd64.appimage, filename-arm64.appimage)
Exactly.
what about the Go version? https://github.com/probonopd/go-appimage/releases/tag/continuous
That repository provides tools to make AppImages which are written in Go. The runtime is not written in Go, but is statically linked C.
AppImage is a Linux only thing since it needs fuse library.
Yes. AppImage is for Linux and Linux compatible systems with a working FUSE setup (such as Windows WSL2 and FreeBSD Linuxulator).
Wrapping a compiled Go static binary inside an AppImage container seriously ham-struck its own packaging capability (in terms of facility coverage).
True, Go applications often don't have the same binary packaging issues that applications written in other languages often have. So for simple Go applications AppImage might be overkill. But keep in mind that in addition to the binary itself, an AppImage can hold icons, descriptions, documentation, graphics, fonts, and other assets. So depending on the application, it may be beneficial even for applications written in Go to be shipped as AppImages. (One cold also put the binary and its assets into an archive like a .zip, but then one would have to extract it prior to being able to execute it; something that takes extra time and space especially if the files are large, such as in games.)
I can't simply script the packaging algorithm like how I did for deb, ipk, DOTNET (nupkg), and Chocolately (nupkg) either since AppImage's low-level is quite complicated.
Actually it's not that complicated, especially for statically linked Go executables. If you don't like to use existing tools but want to code it yourself, then here are the steps:
./AppDir
directory./AppDir/AppRun
./AppDir
That's basically it. Happy to provide more details if needed.
That repository provides tools to make AppImages which are written in Go. The runtime is not written in Go, but is statically linked C.
I see. That explains the notice to use the C version.
AppImage might be overkill. But keep in mind that in addition to the binary itself, an AppImage can hold icons, descriptions, documentation, graphics, fonts, and other assets.
This is what I can see from my side as well (as in, I see a web server type application). There is no doubt this is the value AppImage ecosystem.
We should both agree that's the end-user case (consumer) and only speaks for developer side - packager.
If you don't like to use existing tools but want to code it yourself, then here are the steps:
I think that's a misunderstanding. As specified, it is aligned to my finding where it is not possible to do so since the low-level parts (squashfs and runtime) are C ELF so it is similar to Red Hat rpmbuild
or flatpak-builder
. Hence, for Appimage, it's better to stick to Appimagetool
binary executable.
The low-level scripting means something like understanding the engineering specifications at byte-level and then script the packager using only PowerShell + POSIX-type Shell. For Example: to develop the .deb
packager, I have to go through deeply with its engineering spec (https://www.debian.org/doc/debian-policy/index.html) and then script the compiler from scratch (https://github.com/ChewKeanHo/AutomataCI/blob/main/automataCI/services/compilers/deb.sh for UNIX OS and https://github.com/ChewKeanHo/AutomataCI/blob/main/automataCI/services/compilers/deb.ps1 for Windows OS). This is also the same story with Chocolatey's nupkg
(Windows OS) where the DOTNET nupkg
engineering specs specify it's as a zip
package containing a compulsory xml metadata.
If you notice, both MacOS and Windows can package both nupkg
and .deb
without any no issues locally in the event of cross-compilations even they're not the intended end-users.
I'm simply ask for tidying up your supply-chain releases at https://github.com/AppImage/AppImageKit/releases with something like a shell script similar to Rustup RS (quick and easy hack) or upstream to Homebrew (might takes time but 100% definitely recommended). Both methods emphasize on 1 thing:
appimagekit
and systematically source the tools to a local build system.Here's a case: this upcoming Rust patch's rustup installer is a copy of their rustup.rs
. I'm only importing and review that script; not assembling their installers on-behalf into my own rustup.rs
.
This is the last hurdle. Once resolved, I will develop the integration.
something like a shell script similar to Rustup RS (quick and easy hack)
Good idea: https://github.com/ChewKeanHo/AutomataCI/issues/141
I'm simply ask for tidying up your supply-chain releases at https://github.com/AppImage/AppImageKit/releases with something like a shell script similar to Rustup RS (quick and easy hack) or upstream to Homebrew (might takes time but 100% definitely recommended). Both methods emphasize on 1 thing:
Actually, the version that will become the main one is at https://github.com/AppImage/appimagetool/releases.
I am not familiar with Rustup RS or what it does.
So you need a shell script that downloads the latest version of appimagetool and the runtime?
So you need a shell script that downloads the latest version of linuxdeployqt and the runtime?
Nope. It's for downloading the toolkit from https://github.com/AppImage/appimagetool/releases. Will stick to your main documentation.
I am not familiar with Rustup RS or what it does.
The rustup.rs
is actually a BASH script. A version of it is available here: https://github.com/ChewKeanHo/AutomataCI/blob/experimental/automataCI/services/compilers/rust-rustup.sh
You don't have to mess with curl
's TLS like they did (personally I think they over complicated things with their evangelism). Most important tasks are curl/wget
download and shasum (256/512) against the payload before use. That's why it has to be originated from the source because the shasum values are embedded in it.
If a 2-steps shell scripts are preferred. As in:
I can work on the version-constant ones. So you only work on the version-specific shell script.
I can share with you some POSIX shell codes that can speeds up the development:
SHASUM::create_file()
) - https://github.com/ChewKeanHo/AutomataCI/blob/experimental/automataCI/services/checksum/shasum.shFeel free to copy paste over and assemble yours. =)
Please remember to change the PROJECT_OS
and PROJECT_ARCH
to other naming conventions to avoid conflicts. TQ
Yes, sorry, I meant appimagetool
.
As far as I am concerned, the latest continuous build is the supported one. (Maybe you feel strongly that there should releases be made, but be aware that this is developed by a very small team and we don't have the manpower to support various branches and releases unless a volunteer wants to do it.)
So, something like this?
ARCH=x86_64 # aarch64, armhf, i686, x86_64
wget -c https://github.com/$(wget -q https://github.com/AppImage/appimagetool/releases/expanded_assets/continuous -O - | grep "appimagetool-${ARCH}.AppImage" | head -n 1 | cut -d '"' -f 2)
chmod +x appimagetool-*.AppImage
# TODO: Somehow get a checksum from the GitHub API and compare it to the checksum of the downloaded file
# Then do the same for the runtime from https://github.com/AppImage/type2-runtime
Erms, this is what I meant (handcrafted by myself but can be automated) Lv2 installer script:
appimage-installer-UPDATED.txt
A few things to note:
shasum
values (I use sha256
) and target
filename based on what you support. Then append the rest like a static file (Indicated by: # AUTOMATION - APPEND THE FOLLOWING TO THE ABOVE
).shasum
algorithm are easily updatable (e.g. 256
to 512
) in case of quantum cryptanalysis. Since we're using shasum
external program, it should be fine I guess.dash
capable) in case of future expansion to macOS
or windows
, if technically possible.OS
to the shasum query function just in case the macOS
(darwin
) folks uses it.Try it yourself locally (please do remember to update the shasum values before use).
OK, now I understand better what you are looking for.
I see that there are hardcoded sha256 sums in the script. What I don't understand is how those are supposed to be updated:
Which one do you prefer? Are there better options? It looks like the GitHub API doesn't provide checksums for release assets (details).
You could host this script, but then you would have to update the sha256 sums whenever a new version of linuxdeployqt comes out
This is layer 1 which is a manual audit method.
As I mentioned earlier, don't worry about this. You just have to publish the Layer 2 installer script and I will deal with Level 1 (I roughly have an idea how to do it autonomously; being tracked under Issue #141).
We could release this script together with the linuxdeployqt binaries, with the correct sha256 sums. But then the question is, wouldn't one have to check the checksum of the script itself also?
This is layer 2. Yes, please proceed to host it together. This is definitely the right direction because dealing with 1 installer script is WAY better dealing with a combinations of artifacts for a product line for both sides. It decouples both of our build processes.
To answer your question accurately but IMO a bit too much to ask: you can try certify the installer script itself (e.g. publish GPG armored public key + GPG armored detach signature certificate). I implemented this method by default in this facility where GPG cryptography math don't lies and it makes the entire process a breeze (e.g. automatically trust your installer script and its shasum values once it passed GPG verification) because:
FYI, you're asking a painful industrial problem when you expanding the subject to supporting OSes to Windows (confusing EV Certificates) and MacOS (proprietary Mac-only notarizer).
It looks like the GitHub API doesn't provide checksums for release assets
It's part of your internal build process actually; just shasum the artifact right after its successful compilation and dump the value to a file then at the end of the build cycle, generate the installer script header + append the rest.
Description
Continuation from https://github.com/ChewKeanHo/AutomataCI/issues/126 by @corygalyna
Happy to help but the old forum has been replaced with the new AppImage forum. Please do ask questions there and our community will do our best to support you. Thanks for your kind understanding.
Expected Behavior
AppImage is available.
Current Behavior
AppImage is not available.
Associated Data Files
No response