Graphical programs in Toolpacks. Roadmap/Plans

xplshn commented 2 months ago

Hi. Have you considered adding graphical programs to Toolpacks by means of using type II (their new Go runtime) AppImages? Or maybe even AppBundles (which are simpler than creating AppImages because you its literally as easy as executing a command to create one)?

Else, the other method would be to compile them statically, but the majority of graphical programs tend to have a lot of dependencies, and working around that will be problematic. Another solution would be to use the Nix package manager (their Musl repo, not the Glibc one) and then create a bundle of the executable using PELF or POX by u-root, AppImages could be considered as well, the problem with AppImages is the dependency on Squashfs.

The generation is not the biggest problem, you can generate a wezterm (graphical, not the mux thing) with a simple one-liner. Same for more complex programs like MPV, and if size is a constrain, there exists pelf_small, and also, I can create a new pelf "edition" that contains zstd embedded within it, or a multi-threaded solution like pigz. doas apk add wezterm && dbin run --transparent pelf --main-bin /usr/bin/wezterm --output-to ./wezterm.AppBundle --add-binary /usr/bin/wezterm-gui --add-metadata /usr/share/icons/hicolor/128x128/apps/org.wezfurlong.wezterm.png --add-metadata /usr/share/applications/org.wezfurlong.wezterm.desktop ; doas apk del wezterm

To make the generated bundle also work on Glibc when created from a Musl system, it is simply a matter of adding: --add-ld-and-libc /lib/ld-musl-x86_64.so.1 /lib/libc.musl-x86_64.so.1.

I hope you give this idea a try, a new subdirectory named Bundles could be added, and it could be subdivided as Bundles/* and Bundles/Graphical/*. I have no problem providing one-liners that use apk from Alpine and PELF for this purpose :)

just imagine the possibilities, a completely static Linux generated parting from Toybox's mkroot, and then using `dbin` to add a WM such as `sway` (with libudev-zero contained within it, even pipewire and such)... Its certainly possible, because I've done it before, but manually generating the bundles myself.

Thanks!

xplshn commented 2 months ago

wtf

Azathothas commented 2 months ago

wtf

DO NOT CLICK these links or DOWNLOAD/RUN anything. These are malware meant to backdoor/infect our systems so they can backdoor/infect every binary in the toolpacks. The best response is to ignore them and report to Github. (I have already done so)

As for your actual Issue/Discussion, I will look into this detail later & respond

xplshn commented 2 months ago

wtf

DO NOT CLICK these links or DOWNLOAD/RUN anything. These are malware meant to backdoor/infect our systems so they can backdoor/infect every binary in the toolpacks. The best response is to ignore them and report to Github. (I have already done so)

Yes, I reported them too, not like whatever virus they submitted would work under Musl tho, unless they made it static :)

As for your actual Issue/Discussion, I will look into this detail later & respond

:)

I already convert lots of Alpine packages to .AppBundles, if you want me to share my one-liner to generate a usable self-contained sway WM (includes wezterm and wmenu as well as notification daemon and such), and the rest of one-liners, I can do so at any moment :) The only things I hate to figure out, is if QT programs and programs which use schemas work. I've added support for using AppRuns/AppDirs as well, so that people can use things such as linuxdeployqt and so on, but I've never used linuxdeploy* programs. So I am not sure.

xplshn commented 2 months ago

NOTE: Creating appdirs/appimages with appimagetool from go-appimage & linuxdeploy + gtk/qt plugins is apparently very easy. I just created one of thunar (xfce file manager). These are viable, can be entirely self-contained (-s for selfcontained/static), and they are also not jank made in SH like .AppBundles :)

As for POX, its great as well. I am working on a POX/PELF hybrid that supports the AppDir format as well. So that POX can be used to generate what is basically an AppImage but with code that is actually readable.

Azathothas commented 2 months ago

Hmm, App images. Could also add flatpaks, Debs, rpms, apks I have avoided expanding the windows directory because a lot of the exe turned out to be installers. With expansion to everything and not just static binaries, would have no constraints on what could be added.

xplshn commented 2 months ago

Hmm, App images. Could also add flatpaks, Debs, rpms, apks I have avoided expanding the windows directory because a lot of the exe turned out to be installers. With expansion to everything and not just static binaries, would have no constraints on what could be added.

No, I mean, binaries, appimages compiled with the -s flag are static, and they are self-contained one-click to run programs. They fit very well with the premise of this repo and the premise of dbin. I don't think we need yet another deb/apk/rpm/flatpak/etc repo

Azathothas commented 2 months ago

No worries, anything that's not a static binary won't appear in the metadata.json, thus ensuring dbin works as expected.

I meant to say, that static binaries for me have always meant that they run from a terminal, headless & without any graphics. So the idea of static GUI binaries is new to me. So if do indeed start to add appimages and the like, I will very likely also extend that to apks, exes, debs etc. There are too many buttons to click, too many pages to go through, too many red tapes (some even need accounts etc.), so having a appimage/apk/deb/flatpak/exe/rpm hub isn't necessarily a bad idea.

xplshn commented 2 months ago

No worries, anything that's not a static binary won't appear in the metadata.json, thus ensuring dbin works as expected.

I meant to say, that static binaries for me have always meant that they run from a terminal, headless & without any graphics. So the idea of static GUI binaries is new to me. So if do indeed start to add appimages and the like, I will very likely also extend that to apks, exes, debs etc. There are too many buttons to click, too many pages to go through, too many red tapes (some even need accounts etc.), so having a appimage/apk/deb/flatpak/exe/rpm hub isn't necessarily a bad idea.

I meant to say, that static binaries for me have always meant that they run from a terminal, headless & without any graphics.

That's not true tho, static binaries are usually not used for GUI because GUI toolkits rely too heavily on dynamically linked libraries, however, on Plan9 and OasisLinux, all binaries are static, even graphic ones. Its not an imposibility, its just difficult, but I propose we use .AppImages/.AppBundles and true static binaries where applicable.

I will very likely also extend that to apks, exes, debs etc.

One question I have to ask is; why? What's the point? I thought people wanted static binaries because they're not packages, they're simple, they're reliable, they don't even have to be installed and the location from which you execute them does not appear as well. Also, note that you'd have to sign such binaries, and it does not make much sense because all binaries you package here as static are already in the repos of large distributions, people would have to go out of their way to specifically add your repo.

There are too many buttons to click, too many pages to go through.

What do you mean?

The proposal was simple; Add small X11/Wayland programs using PELF and Alpine, some other complex programs can be provided as AppImages (appimages require writting an apprun or using a deploy tool)

xplshn commented 2 months ago

This is if you want to generate an AppBundle:

pelf --main-bin /usr/bin/mpv --output-to ./mpv.AppBundle --add-metadata /usr/share/applications/mpv.desktop --add-metadata /usr/share/icons/hicolor/128x128/apps/mpv.png --add-ld-and-libc /lib/ld-musl-x86_64.so.1 /lib/libc.musl-x86_64.so.1

Note, however, that it is best-practice to replace the exec= lines from exec=binaryName to exec=binaryName.AppBundle. So that pelfd doesn't have to do this.

Another example, but for wezterm:

pelf --main-bin /usr/bin/wezterm --add-binary /usr/bin/wezterm-gui --output-to ./wezterm.AppBundle
--add-metadata /usr/share/applications/org.wezfurlong.wezterm.desktop --add-metadata /usr/share/icons/hicolor/128x128/apps/org.wezfurlong.wezterm.png --add-ld-and-libc /lib/ld-musl-x86_64.so.1 /lib/libc.musl-x86_64.so.1

xplshn commented 2 months ago

mpv uses dlopen() to load DRI firmare/mesa. So, expect problems if running on a headless environment. There are better examples for AppBundles/AppImages than mpv.

Samueru-sama commented 2 months ago

Check my anylinux mpv appimage: https://github.com/Samueru-sama/mpv-AppImage/releases/tag/continuous

It uses the deploy everything mode of go-appimage, and MPV is build using the mpv-build scripts which statically link a lot of stuff.

However it is not perfect, for some reason opening videos with yt-dlp doesn't work, but besides that it has no problem playing local videos.

And it seems the issue with yt-dlp is related to glibc being bundled, so it is likely that running the build scripts on a musl enviroment might fix that issue, however it is also likely that the script might not work on alpine lol

Another appimage that uses the deploy everything mode is the flameshot appimage.

xplshn commented 2 months ago

Check my anylinux mpv appimage: https://github.com/Samueru-sama/mpv-AppImage/releases/tag/continuous

It uses the deploy everything mode of go-appimage, and MPV is build using the mpv-build scripts which statically link a lot of stuff.

However it is not perfect, for some reason opening videos with yt-dlp doesn't work, but besides that it has no problem playing local videos.

And it seems the issue with yt-dlp is related to glibc being bundled, so it is likely that running the build scripts on a musl enviroment might fix that issue, however it is also likely that the script might not work on alpine lol

Another appimage that uses the deploy everything mode is the flameshot appimage.

You could refactor a bit to use this: https://github.com/jirutka/setup-alpine

Samueru-sama commented 2 months ago

Check my anylinux mpv appimage: https://github.com/Samueru-sama/mpv-AppImage/releases/tag/continuous It uses the deploy everything mode of go-appimage, and MPV is build using the mpv-build scripts which statically link a lot of stuff. However it is not perfect, for some reason opening videos with yt-dlp doesn't work, but besides that it has no problem playing local videos. And it seems the issue with yt-dlp is related to glibc being bundled, so it is likely that running the build scripts on a musl enviroment might fix that issue, however it is also likely that the script might not work on alpine lol Another appimage that uses the deploy everything mode is the flameshot appimage.

You could refactor a bit to use this: https://github.com/jirutka/setup-alpine

I've tested making the appimage in an alpine container, after a lot of pain I was able to get it to launch.

However the issue of mpv not playing youtube videos is still there, this is the error that I get:

./mpv_Media_Player-v0.38.0-743-gad7976c33e-x86_64.AppImage "https://www.youtube.com/watch?v=dQw4w9WgXcQ"                                                                                                                                                                      4907ms 
No protocol handler found to open URL https://www.youtube.com/watch?v=dQw4w9WgXcQ
The protocol is either unsupported, or was disabled at compile-time.
No protocol handler found to open URL https://rr3---sn-hp57yn7r.googlevideo.com/videoplayback?expire=1725438108&ei=PMTXZpPKA4-my_sPge_LuAQ&ip=38.25.175.104&id=o-AIAyJlPZc-Ur8AiUlyyhe8dzIjV2azYmM7urQAvKo__U&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=7c&mm=31%2C29&mn=sn-hp57yn7r%2Csn-hp57knds&ms=au%2Crdu&mv=m&mvi=3&pl=19&initcwndbps=722500&vprv=1&svpuc=1&mime=audio%2Fwebm&rqh=1&gir=yes&clen=3437753&dur=212.061&lmt=1717047822556748&mt=1725416221&fvip=5&keepalive=yes&c=IOS&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswQwIgBR5_cmGqM5IA5M86a-9YS7uXNJ8Sf55bHZl7J-1GZWACHxUcqulIoOmKwUZdO5KoIs3sjmhSiImNYt4KTxxgeEA%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABPmVW0wRgIhAPS3Crcf4LgLNnB1z5OOW3apfmmPNzdNU5DJvnIdXj0_AiEA6GEarvWkWLuPGLV1ZCfVcFi_kPf9S_mlFvwt2RJYZZ8%3D
The protocol is either unsupported, or was disabled at compile-time.
EDL: Could not open source file 'https://rr3---sn-hp57yn7r.googlevideo.com/videoplayback?expire=1725438108&ei=PMTXZpPKA4-my_sPge_LuAQ&ip=38.25.175.104&id=o-AIAyJlPZc-Ur8AiUlyyhe8dzIjV2azYmM7urQAvKo__U&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=7c&mm=31%2C29&mn=sn-hp57yn7r%2Csn-hp57knds&ms=au%2Crdu&mv=m&mvi=3&pl=19&initcwndbps=722500&vprv=1&svpuc=1&mime=audio%2Fwebm&rqh=1&gir=yes&clen=3437753&dur=212.061&lmt=1717047822556748&mt=1725416221&fvip=5&keepalive=yes&c=IOS&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswQwIgBR5_cmGqM5IA5M86a-9YS7uXNJ8Sf55bHZl7J-1GZWACHxUcqulIoOmKwUZdO5KoIs3sjmhSiImNYt4KTxxgeEA%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABPmVW0wRgIhAPS3Crcf4LgLNnB1z5OOW3apfmmPNzdNU5DJvnIdXj0_AiEA6GEarvWkWLuPGLV1ZCfVcFi_kPf9S_mlFvwt2RJYZZ8%3D'.
No video or audio streams selected.
Exiting... (Errors when loading file)

this only happens when the deploy everything mode is used, which bundles glibc/musl into the appimage, if it doesn't get used this error doesn't happen, however that means that appimage isn't as universal as a static binary.

edit: I think the issue with mpv is that I'm missing a library that is no seen by ldd since mpv uses yt-dlp to open youtube videos.

xplshn commented 2 months ago

Check my anylinux mpv appimage: https://github.com/Samueru-sama/mpv-AppImage/releases/tag/continuous It uses the deploy everything mode of go-appimage, and MPV is build using the mpv-build scripts which statically link a lot of stuff. However it is not perfect, for some reason opening videos with yt-dlp doesn't work, but besides that it has no problem playing local videos. And it seems the issue with yt-dlp is related to glibc being bundled, so it is likely that running the build scripts on a musl enviroment might fix that issue, however it is also likely that the script might not work on alpine lol Another appimage that uses the deploy everything mode is the flameshot appimage.

You could refactor a bit to use this: https://github.com/jirutka/setup-alpine

I've tested making the appimage in an alpine container, after a lot of pain I was able to get it to launch.

However the issue of mpv not playing youtube videos is still there, this is the error that I get:
./mpv_Media_Player-v0.38.0-743-gad7976c33e-x86_64.AppImage "https://www.youtube.com/watch?v=dQw4w9WgXcQ"                                                                                                                                                                      4907ms 
No protocol handler found to open URL https://www.youtube.com/watch?v=dQw4w9WgXcQ
The protocol is either unsupported, or was disabled at compile-time.
No protocol handler found to open URL https://rr3---sn-hp57yn7r.googlevideo.com/videoplayback?expire=1725438108&ei=PMTXZpPKA4-my_sPge_LuAQ&ip=38.25.175.104&id=o-AIAyJlPZc-Ur8AiUlyyhe8dzIjV2azYmM7urQAvKo__U&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=7c&mm=31%2C29&mn=sn-hp57yn7r%2Csn-hp57knds&ms=au%2Crdu&mv=m&mvi=3&pl=19&initcwndbps=722500&vprv=1&svpuc=1&mime=audio%2Fwebm&rqh=1&gir=yes&clen=3437753&dur=212.061&lmt=1717047822556748&mt=1725416221&fvip=5&keepalive=yes&c=IOS&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswQwIgBR5_cmGqM5IA5M86a-9YS7uXNJ8Sf55bHZl7J-1GZWACHxUcqulIoOmKwUZdO5KoIs3sjmhSiImNYt4KTxxgeEA%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABPmVW0wRgIhAPS3Crcf4LgLNnB1z5OOW3apfmmPNzdNU5DJvnIdXj0_AiEA6GEarvWkWLuPGLV1ZCfVcFi_kPf9S_mlFvwt2RJYZZ8%3D
The protocol is either unsupported, or was disabled at compile-time.
EDL: Could not open source file 'https://rr3---sn-hp57yn7r.googlevideo.com/videoplayback?expire=1725438108&ei=PMTXZpPKA4-my_sPge_LuAQ&ip=38.25.175.104&id=o-AIAyJlPZc-Ur8AiUlyyhe8dzIjV2azYmM7urQAvKo__U&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=7c&mm=31%2C29&mn=sn-hp57yn7r%2Csn-hp57knds&ms=au%2Crdu&mv=m&mvi=3&pl=19&initcwndbps=722500&vprv=1&svpuc=1&mime=audio%2Fwebm&rqh=1&gir=yes&clen=3437753&dur=212.061&lmt=1717047822556748&mt=1725416221&fvip=5&keepalive=yes&c=IOS&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswQwIgBR5_cmGqM5IA5M86a-9YS7uXNJ8Sf55bHZl7J-1GZWACHxUcqulIoOmKwUZdO5KoIs3sjmhSiImNYt4KTxxgeEA%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABPmVW0wRgIhAPS3Crcf4LgLNnB1z5OOW3apfmmPNzdNU5DJvnIdXj0_AiEA6GEarvWkWLuPGLV1ZCfVcFi_kPf9S_mlFvwt2RJYZZ8%3D'.
No video or audio streams selected.
Exiting... (Errors when loading file)
this only happens when the deploy everything mode is used, which bundles glibc/musl into the appimage, if it doesn't get used this error doesn't happen, however that means that appimage isn't as universal as a static binary.

edit: I think the issue with mpv is that I'm missing a library that is no seen by ldd since mpv uses yt-dlp to open youtube videos.

Have you tried to do the following:

Get yt-dlp from pip3 (use --user, to avoid destroying the system)
Get pyinstaller
Use the -f option to generate a single file yt-dlp binary
Add the yt-dlp single-file binary to the appimage (no need to use staticx since the appimage contains both ld.so and libc.so)

Azathothas commented 2 months ago

As discussed in https://github.com/Samueru-sama/mpv-AppImage/issues/2#issuecomment-2333165714

The current, most logical approach is to use https://github.com/ralismark/nix-appimage along with some custom hook (WIP, to reduce the size but not by much, probably just 5% or less) to generate the most portable appimage possible. The only downside being size, if there's any other do let me know now. If this is acceptable (Universal Portable AppImages but Gigantic in size), please react with 👍 otherwise react with 👎 If there's no significant backlash, then I will go ahead and invest time in this. This is so promising that I think we can exceed the merely ~ 1500 AppImages hosted at https://www.appimagehub.com/ in no time with the added benefit of it working everywhere but the added downside of being large in size.

Samueru-sama commented 2 months ago

As discussed in Samueru-sama/mpv-AppImage#2 (comment)

The current, most logical approach is to use https://github.com/ralismark/nix-appimage along with some custom hook (WIP, to reduce the size but not by much, probably just 5% or less) to generate the most portable appimage possible. The only downside being size, if there's any other do let me know now. If this is acceptable (Universal Portable AppImages but Gigantic in size), please react with 👍 otherwise react with 👎 If there's no significant backlash, then I will go ahead and invest time in this. This is so promising that I think we can exceed the merely ~ 1500 AppImages hosted at https://www.appimagehub.com/ in no time with the added benefit of it working everywhere but the added downside of being large in size.

I'm fine I don't think it is worth it for simple apps like mpv (because the waste of storage is massive relative to the application).

But for apps like Steam, Gimp, it would be interesting to see the results.

Azathothas commented 2 months ago

I'm fine I don't think it is worth it for simple apps like mpv (because the waste of storage is massive relative to the application).

All the binaries I tested for mpv were broken in some way, and if it's similar case for most other publically available appimages then I don't think it's not worth the size.

Ofc, I will accept PRs that attempt to create similarly portable AppImages manually without nix-appimage.

xplshn commented 2 months ago

As discussed in Samueru-sama/mpv-AppImage#2 (comment)

The current, most logical approach is to use https://github.com/ralismark/nix-appimage along with some custom hook (WIP, to reduce the size but not by much, probably just 5% or less) to generate the most portable appimage possible. The only downside being size, if there's any other do let me know now. If this is acceptable (Universal Portable AppImages but Gigantic in size), please react with 👍 otherwise react with 👎 If there's no significant backlash, then I will go ahead and invest time in this. This is so promising that I think we can exceed the merely ~ 1500 AppImages hosted at https://www.appimagehub.com/ in no time with the added benefit of it working everywhere but the added downside of being large in size.

I am very much interested, about the hooks you talk about. What about using upx for processing the binaries inside of the nix stores? If size goes down, download speed and initial execution time will also be faster (for those with old hardware, like me). I don't really like nix being an everything-solution to package management, system configuration, and etc, but I'll give it a try on my Athlon PC.

Other hook scripts could be: Removing systemd, removing anything that says "example", removing locales support, etc. Or, using Musl will make tha AppImage 30% smaller, and we can make it lighter as well.

Azathothas commented 2 months ago

Using upx breaks everything. It simply doesn't work (sometimes segfaults, sometimes other errors) Even if it worked, upx binaries have to unload themselves in memory at runtime, and if dozens of binaries are all upxed and need to be unpacked and on top of the appimage format itself, it's a massive penalty in terms of startup speed & performance.

Removing systemd, examples, tests, node modules, docs/man pages, etc seemed trivial at first. But all nix stores are different, and we would have to have a hundred if else checks just to reduce a few MBs, not to mention the cost to create an appimage in terms of time and effort would be too much. And at any moment, the nix-pkg could change in terms of layout & behaviour and we would have to redo the process again. nix build has limited support for musl, after all, if a package gets built with the pkgStatic attribute, then we already have a static binary, no need for app images. nix-appimages is very very fast, compiling complex apps in minutes, because it fetches a lot of prebuilt artifacts from nix's cache. Using musl will mean rebuild everything, the process would be similar to bootstrapping an entire os/system each time for each package. At that point, manually building/bundling is much simpler.

xplshn commented 2 months ago

Removing systemd, examples, tests, node modules, docs/man pages, etc seemed trivial at first. But all nix stores are different, and we would have to have a hundred if else checks just to reduce a few MBs, not to mention the cost to create an appimage in terms of time and effort would be too much. And at any moment, the nix-pkg could change in terms of layout & behaviour and we would have to redo the process again.

I am not suggesting that we do this in a "smart" way at all, just an rm -rf over directories that are known to be unnecesary. Also, removing unnecesary default packages, such as bash from the nix-generated appimage.

nix build has limited support for musl, after all, if a package gets built with the pkgStatic attribute, then we already have a static binary, no need for app images.

Musl is a libc, not a tool/means of building a static binary, if they offer some binaries as Musl, such as Firefox, it'd be best to use that instead of the glibc one.

xplshn commented 2 months ago

Ajam, the binaries from Oasis are fantastic, we could borrow the entire build system after renaming /share to /usr/share so that the binaries work on generic Linux systems

This netsurf in temp/ weights about 123.8 MiB, while the static one built by OasisLinux is just 4.1 megs

xplshn commented 2 months ago

Yes, a graphical web browser that is lighter than Elinks, and its lighter than my text editor (micro).

xplshn commented 2 months ago

Let's not waste the effort Michael Forney put into making reliable recipes to build lightweight static binaries for Oasis, what do you think? These could become ppkg formulas as well. I'll try to create a recipe based off of Oasis/pkg ones

Samueru-sama commented 2 months ago

I am not suggesting that we do this in a "smart" way at all, just an rm -rf over directories that are known to be unnecesary. Also, removing unnecesary default packages, such as bash from the nix-generated appimage.

Indeed, we've recently reduced the size of the Steam appimage by removing all the static libraries.

Originally that appimage was 800 MiB and was lacking some features, we've added more stuff and did a bunch of rm -rf and now the size is 544 MiB.

The Steam appimage made from nix is 1.2 GiB and doesn't work 😭

Azathothas commented 2 months ago

Oasis'a mpv didn't work when @Samueru-sama tried using it so I didn't have high hopes. Regardless if their binaries do work, I would need a build recipe , and looking at their GitHub page and issues (including ones you made) I don't think it's so easy. If you can write a build script that can build the binaries from their repo, sure I will take a look.

This is mostly a beta test. And I thought I had sacrifice small size for maximum portability and easiest creation. But if these appimage don't work, there's no point in pursuing this. There's much work already done at Appimage-Manager(AM) so I may actually close this issue as not planned.

Azathothas commented 2 months ago

I tried purging things like llvm and such yesterday, but the appimage refused to run. I tried with several of them and as soon as I had delete something, it would break the entire appimage. That's why I didn't think rm rf was wise. But I am going to take another look at this again today

Azathothas commented 2 months ago

So finally after hours of testing by trial and error, this is what I have come up with which works in almost all (still some exceptions) of the nix-generated AppImages.

!#Deleting Header files
find "." -type d -path "*/include*" -print -exec rm -rf {} 2>/dev/null \; 2>/dev/null

!# Deleting docs & manpages
find "." -type d -path "*doc/share*" ! -name "*${PKG_NAME}*" -print -exec rm -rf {} 2>/dev/null \; 2>/dev/null
find "." -type d -path "*/share/docs*" ! -name "*${PKG_NAME}*" -print -exec rm -rf {} 2>/dev/null \; 2>/dev/null
find "." -type d -path "*/share/man*" ! -name "*${PKG_NAME}*" -print -exec rm -rf {} 2>/dev/null \; 2>/dev/null

!#Deleting static libs (.a files)
find "." -type f -name "*.a" -print -exec rm -f {} 2>/dev/null \; 2>/dev/null

!#Turns out that systemd .so files are needed, so other than those, everything else (service files etc) gets pruned
find "." -type d -name "*systemd*" -exec find {} -type f ! -name "*.so*" -delete \;

!#I tried Stripping the binaries (& libs) but this also breaks things
#find "." -type f -executable -exec file -i '{}' \; | grep "application/.*executable" | cut -d':' -f1 | xargs realpath | xargs sudo strip -R ".comment" -R ".gnu.version" --strip-unneeded 2>/dev/null

It reduces the size by 40-50% (Exceptions apply)

!#Examples
ladybird.AppImage (Nix) [388 MB]  --> (Compressed & Repacked) 312 MB --> (Debloated) 261 MB ! 32% !
- [+] https://pub.ajam.dev/temp/ladybird.AppImage
mpv.AppImage (Nix) [450 MB] --> (Compressed & Repacked) 357 MB --> (Debloated) 257 MB ! 42% !
- [+] https://pub.ajam.dev/temp/mpv.AppImage
netsurf.AppImage (Nix) [158M MB] --> (Compressed & Repacked) 123 MB --> (Debloated) 115MB ! 27% !
- [+] https://pub.ajam.dev/temp/netsurf.AppImage

The AppImages are still bloated in size but, I think, that's the most that can be done for now without completely breaking the AppImage. I referenced, https://github.com/ivan-hc/Steam-appimage/blob/bae942d82a88ab8b1acb35c46f71d1c0564b95c2/create-arch-bootstrap.sh#L380 and @Samueru-sama helped me extensively. If there's something else that can be done, do let me know now, as it will be difficult to change the scripts after there's more of them.

Azathothas commented 2 months ago

I have also changed the steam appimage script to directly fetch from @ivan-hc repo: https://github.com/ivan-hc/Steam-appimage https://github.com/Azathothas/Toolpacks/blob/main/.github/scripts/x86_64_Linux/bins/steam.sh

xplshn commented 2 months ago

@Azathothas None of the AppImages currently on the repo work on Alpine. Only Helix worked. All of them whine about libEGL. I tried those marked with [c] (cached, because I ran them with dbin run)

Azathothas commented 2 months ago

Oh no, if these appimages don't work on musl then there's no point in adding them. Could you try installing mesa egl and try again.

And how did netsurf work yesterday, or it didn't work...

Edit: Try the ones from https://pub.ajam.dev/temp/ , the ones in the repo are from yesterday, they are outdated

xplshn commented 2 months ago

The netsurf from OasisLinux worked after adding /share to my system. The one here didn't

Azathothas commented 2 months ago

So you are saying even after installing mesa egl , none of the app images work in musl?

I should have tested this sooner, my test environment is just debian (headless and one with xfce), so I didn't think it would only work on glibc. But it doesn't make sense, these app images are bundled with glibc so they should work anywhere. Are you sure your system already contains the libegl

If it doesn't work, then I think all the of the work was a waste. I won't add appimages that only work on glibc.

xplshn commented 2 months ago

But it doesn't make sense, these app images are bundled with glibc so they should work anywhere.

Should have started with Musl, Musl's proved to work containarized, while Glibc always has issues

Azathothas commented 2 months ago

Try renaming mpv vhsgunzo to just mpv It's like BusyBox you can either renaame the main binary or create symlinks, it should work, it was made on void

As for others, let's see what @Samueru-sama has to say. It's the Weeknd so I might not be available for a bit. But yeah if nix made appimages don't work, then the only alternative would be to use run image https://github.com/VHSgunzo/runimage or to abandon this format altogether

xplshn commented 2 months ago

Why not use the Anylinux releases that Samu makes? All of his repos are using Github Actions as CIs to build the AppImages.

xplshn commented 2 months ago

Azathothas commented 2 months ago

So it seems like it's mostly my fault for being too optimistic. Had I read the description of https://github.com/ralismark/nix-appimage?tab=readme-ov-file#caveats , I would have been aware of: https://github.com/NixOS/nixpkgs/issues/9415 https://github.com/AppImage/AppImageKit/issues/877

xplshn commented 2 months ago

We should prioritize quality over quantity... Adding instead the AppImages that are known to work in both Glibc & Musl. And I'd rather also we use eget to download from the CI runs directly instead of building them ourselves if possible.

Samueru-sama commented 2 months ago

So it seems like it's mostly my fault for being too optimistic. Had I read the description of https://github.com/ralismark/nix-appimage?tab=readme-ov-file#caveats , I would have been aware of: NixOS/nixpkgs#9415 AppImage/AppImageKit#877

The second issue is already solved btw, just that appimagekit (which we don't use) hasn't updated to use the static runtime, the appimagetool that is being used already uses the static runtime by default.

Also I did have EGL erors when testing the nix appimages, but they still worked.

Azathothas commented 2 months ago

Currently evaluating runimage, check https://t.me/VHSgroup/22859 for the progress.

AppImages won't be created/supported (maybe a dozen here there) until at least https://github.com/NixOS/nixpkgs/issues/9415 is fixed or https://github.com/nix-community/nixGL is itself added/merged to https://github.com/ralismark/nix-appimage There's also this (the reason why Steam didn't work): https://github.com/ralismark/nix-appimage/issues/10

AppBundles had similar issues & as confirmed by the author, it will be going through a major rewrite, so currently it's not suited.

The potential solution, whatever it may be, must be portable (don't depend on the host) & cross-libc (musl & glibc).

I recently read this blog: https://wiki.qt.io/QtCS2024_Compile_once._Run_everywhere https://git.qt.io/cradam/presentations/-/raw/main/2024.09.06/QtCS2024-CompileOnce-RunEverywhere.pdf?ref_type=heads HN: https://news.ycombinator.com/item?id=41470571 They use cosmopolitan. However it seems like each and every ported pkg requires some manual patching, so this means tremendous cost in porting anything. But this would also help us create binaries that are not only cross libc but also cross-platform (BSDs, Unix, Windows, & Mac)

xplshn commented 2 months ago

Currently evaluating runimage, check https://t.me/VHSgroup/22859 for the progress.

AppImages won't be created/supported (maybe a dozen here there) until at least NixOS/nixpkgs#9415 is fixed or https://github.com/nix-community/nixGL is itself added/merged to https://github.com/ralismark/nix-appimage There's also this (the reason why Steam didn't work): ralismark/nix-appimage#10

AppBundles had similar issues & as confirmed by the author, it will be going through a major rewrite, so currently it's not suited.

The potential solution, whatever it may be, must be portable (don't depend on the host) & cross-libc (musl & glibc).

I recently read this blog: https://wiki.qt.io/QtCS2024_Compile_once._Run_everywhere https://git.qt.io/cradam/presentations/-/raw/main/2024.09.06/QtCS2024-CompileOnce-RunEverywhere.pdf?ref_type=heads HN: https://news.ycombinator.com/item?id=41470571 They use cosmopolitan. However it seems like each and every ported pkg requires some manual patching, so this means tremendous cost in porting anything. But this would also help us create binaries that are not only cross libc but also cross-platform (BSDs, Unix, Windows, & Mac)

Superconfigure allows you to build many programs with Cosmopolitan, its amazing, check it out here: https://github.com/ahgamut/superconfigure/

Also, if you end up adding Cosmopolitan to the repo, a "generic" arch/dir should be considered.

xplshn commented 2 months ago

The also build a python binary that is portable :)

Azathothas commented 2 months ago

Update: @xplshn seems to have wrongly reported that nix-appimage don't work musl. I installed fresh alpine with xfce and tested the AppImages from https://pub.ajam.dev/temp/ firefox works. ladybird doesn't, seems to be a known issue: https://github.com/LadybirdBrowser/ladybird/issues/371 mpv works. netsurf works. ppsspp, fails because I don't have audio system (& I can't make it work with vmware/hyperv) steam doesn't, also a known issue: https://github.com/ralismark/nix-appimage/issues/10

xplshn commented 2 months ago

@xplshn seems to have wrongly reported that nix-appimage don't work musl.

It did not work on my install of Alpine. I'll test the newer releases in a few hours when I get home

Azathothas commented 1 month ago

Alright, I have decided these:

Have another repo for AppBundles | AppImages | FlatImages | RunImages & any other related format.
- Maybe titled Toolpacks-Extras ? Or maybe as part of an orginazation in case some of the authors of the above mentioned packages/package-managers would like to have it that way. (I have contacted the authors, but it's not guaranteed)
Host the actual binary blobs on Hugging Face as they let us use git-lfs for free without any restrictions.
Have metadata similar to Toolpacks, so other frontends can use it programmatically.

Azathothas commented 1 month ago

Migrated to https://github.com/Azathothas/Toolpacks-Extras

Azathothas / Toolpacks

Graphical programs in Toolpacks. Roadmap/Plans #28