xplshn / pelf

Pack an ELF. Turn your binaries into single-file executables [.AppBundle|.blob], similar to AppImages and compatible with AppDirs. LIBC-independent, works on *BSDs.
BSD 3-Clause "New" or "Revised" License
12 stars 1 forks source link

Support for AppDirs #3

Closed xplshn closed 2 months ago

xplshn commented 2 months ago

@Samueru-sama Sorry to bother you, I got an initial AppDir edition working, what is different about AppImages? How do they make the programs access the shipped files under "usr/"? Any cool tricks to do this? Thanks!

Samueru-sama commented 2 months ago

Usually (99% of the cases) the AppDir contains a script named AppRun that sets a few env variables, including the current location of the program and then starts it.

However access to files in usr/ is done by linuxdeploy as it patches the binary to have its path relative to the usr inside the appimage instead of /usr from the filesystem.

that way it can always find what it needs without needing to use stuff like LD_PRELOAD or LD_LIBRARY_PATH.

You can also change the default behavior of the program this way, for example here is how the AppRun of my mpv appimage looks like:

#!/bin/sh
CURRENTDIR="$(dirname "$(readlink -f "$0")")"
if [ -z "$@" ]; then
    "$CURRENTDIR/usr/bin/mpv" --player-operation-mode=pseudo-gui
else
    "$CURRENTDIR/usr/bin/mpv" "$@"
fi

By default mpv doesn't launch its GUI when you run the binary (double click on the appimage) this change makes it do so.

And here is the AppRun of suyu which is a bit more complex:

#!/bin/sh -e

# A shell script that does the same as the binaries in the release section.

cd "$(dirname "$0")"

cxxpre=""
gccpre=""
execpre=""
libc6arch="libc6,x86-64"
exec="./bin/$(sed -n -e 's|%f||' -e 's|^Exec=||p' $(ls -1 *.desktop))"

if [ -n "$APPIMAGE" ] && [ "$(file -b "$APPIMAGE" | cut -d, -f2)" != " x86-64" ]; then
  libc6arch="libc6"
fi

cd "usr"

if [ -e "./optional/libstdc++/libstdc++.so.6" ]; then
  lib="$(PATH="/sbin:$PATH" ldconfig -p | grep "libstdc++\.so\.6 ($libc6arch)" | awk 'NR==1{print $NF}')"
  sym_sys=$(tr '\0' '\n' < "$lib" | grep -e '^GLIBCXX_3\.4' | sort -V | tail -n1)
  sym_app=$(tr '\0' '\n' < "./optional/libstdc++/libstdc++.so.6" | grep -e '^GLIBCXX_3\.4' | sort -V | tail -n1)
  if [ "$(printf "${sym_sys}\n${sym_app}"| sort -V | tail -1)" != "$sym_sys" ]; then
    cxxpath="./optional/libstdc++:"
  fi
fi

if [ -e "./optional/libgcc/libgcc_s.so.1" ]; then
  lib="$(PATH="/sbin:$PATH" ldconfig -p | grep "libgcc_s\.so\.1 ($libc6arch)" | awk 'NR==1{print $NF}')"
  sym_sys=$(tr '\0' '\n' < "$lib" | grep -e '^GCC_[0-9]\\.[0-9]' | sort -V | tail -n1)
  sym_app=$(tr '\0' '\n' < "./optional/libgcc/libgcc_s.so.1" | grep -e '^GCC_[0-9]\\.[0-9]' | sort -V | tail -n1)
  if [ "$(printf "${sym_sys}\n${sym_app}"| sort -V | tail -1)" != "$sym_sys" ]; then
    gccpath="./optional/libgcc:"
  fi
fi

if [ -n "$cxxpath" ] || [ -n "$gccpath" ]; then
  if [ -e "./optional/exec.so" ]; then
    execpre=""
    export LD_PRELOAD="./optional/exec.so:${LD_PRELOAD}"
  fi
  export LD_LIBRARY_PATH="${cxxpath}${gccpath}${LD_LIBRARY_PATH}"
fi

# Force xcb platform for Qt applications
if [ -z "${QT_QPA_PLATFORM}" ]; then
    export QT_QPA_PLATFORM=xcb
fi

# Find correct root CA file
_POSSIBLE_CERTIFICATES="/etc/ssl/certs/ca-bundle.crt \
/etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt \
/etc/ssl/ca-bundle.pem /etc/pki/tls/cacert.pem"

if [ -z "$SSL_CERT_FILE" ]; then
    for i in $_POSSIBLE_CERTIFICATES; do 
        if [ -f "$i" ]; then
            export SSL_CERT_FILE="$i"
            break
        fi
    done
fi

#echo ">>>>> $LD_LIBRARY_PATH"
#echo ">>>>> $LD_PRELOAD"

exec $exec "$@"
xplshn commented 2 months ago

So, I shouldn't worry because the AppRun supplied by the user already does the handling?

xplshn commented 2 months ago

Any vars that AppRuns expect to be set?

Samueru-sama commented 2 months ago

So, I shouldn't worry because the AppRun supplied by the user already does the handling?

Yes, although the AppRun needs to be created. iirc linuxdeploy-qt and linuxdeploy-gtk do generate some sort of generic AppRun if the user doesn't make one.

And some applications do make their AppRun a binary, those are very very rare though. And sometimes AppRun is just a symlink to the binary inside usr/bin as well.

But at the very minimum the AppRun has to launch the main binary in the application.

Any vars that AppRuns expect to be set?

None.


Are you planing on taking an AppDir and make able to work on BSDs?

You can experiment with this as that AppImage even works on musl distros because it uses the static runtime with a static htop binary, I don't know if the binary itself can work on BSD though.

xplshn commented 2 months ago

Okay. So, it seems like I've already added AppDir support and I didn't even know it :laughing: So, what's left know is handling thumbnails and icon files. The format used by PELF is simple, but AppImages aren't so simple, they pack various resolutions of the icons, so I haven't yet figured out how to handle that.

case "$1" in
    --pbundle_help)
        printf "This bundle was generated automatically by PELF __PELF_VERSION__, the machine on which it was created has the following \"uname -mrspv\":\n %s \n" "__PELF_HOST__"
        printf "Usage:\n <--pbundle_link <binary>|--pbundle_help|--pbundle_xpmIcon|--pbundle_pngIcon|--pbundle_svgIcon|--pbundle_desktop> <args...>\n"
        printf "EnvVars:\n USE_BULKLIBS=[0,1]\n USE_SYSTEM_LIBRARIES=[1,0]\n SHOW_DISCARDPROCESS=[0,1]\n HELP_PAGE_LIST_PACKEDFILES=[0,1]\n REUSE_INSTANCES=[0,1]\n CREATE_THUMBNAILS=[1,0]\n"
        if [ "$HELP_PAGE_LIST_PACKEDFILES" = "1" ]; then
            ls "$TMPDIR"/*
        fi
        exit 1
        ;;
    --pbundle_link)
        binDest="$2"
        shift 2
        ;;
    --pbundle_pngIcon)
        for icon_path in "$TMPDIR/bundledMetadata/icon.png" "$TMPDIR/.DirIcon"; do
            if [ -f "$icon_path" ]; then
                base64 "$icon_path"
                exit 0
            fi
        done
        exit 1
        ;;
    --pbundle_xpmIcon)
        icon_path="$TMPDIR/bundledMetadata/icon.xpm"
        if [ -f "$icon_path" ]; then
            base64 "$icon_path"
            exit 0
        else
            exit 1
        fi
        ;;
    --pbundle_svgIcon)
        icon_path="$TMPDIR/bundledMetadata/icon.svg"
        if [ -f "$icon_path" ]; then
            base64 "$icon_path"
            exit 0
        else
            exit 1
        fi
        ;;
    --pbundle_desktop)
        desktop_path="$TMPDIR/bundledMetadata/app.desktop"
        if [ -f "$desktop_path" ]; then
            base64 "$desktop_path"
            exit 0
        elif desktop_file=$(find "$TMPDIR" -maxdepth 1 -name '*.desktop' -print -quit); then
            if [ -n "$desktop_file" ]; then
                base64 "$desktop_file"
                exit 0
            fi
        else
            exit 1
        fi
        ;;
esac
Samueru-sama commented 2 months ago

Okay. So, it seems like I've already added AppDir support and I didn't even know it 😆 So, what's left know is handling thumbnails and icon files. The format used by PELF is simple, but AppImages aren't so simple, they pack various resolutions of the icons, so I haven't yet figured out how to handle that.

Any appimage made with linuxdeploy or appimagetool or go-appimagetool (aka 99.99% of appimages) will have a .DirIcon file in the top level of the appimage.

Sometimes that file is a symlink and one has to be careful when extracting it. This is how AM handles it

Same applies for the .desktop

xplshn commented 2 months ago

Everything's done already, now I've gotta polish the script a bit. A lot of PELF has been changed, now it receives either --main-bin OR --add-appdir, and it HAS to receive an --output-to argument. After I format the script better, I will try (and probably fail) to generate an appdirs of Caja (Mate's File Manager) to test everything. I've also created a small utility called "getlibs" that basically gets the .so files a binary needs and copies them to a directory. Since I found out that linuxdeploy doesn't do this already, at least not with every dependency.

]/Documents/TrulyMine/pelfx/work @ cat ../cmd/misc/getlibs
#!/bin/sh

add_thelibs() {
 # Copy the libraries from the executable to the temporary directory
 SOs="$(ldd "$1")"
 echo "$SOs" | awk '
     # Store the first word of the first line
     NR == 1 { first_word = $1 }
     # For lines with =>, check if the third word is not the same as the first word of the first line
     /=>/ && $3 != first_word { print $3 }
    '| while read -r lib; do
     # Copy the library to the temporary directory
     cp -L "$lib" "$2" || exit 1
 done
}

add_thelibs "$@"
Samueru-sama commented 2 months ago

I've also created a small utility called "getlibs" that basically gets the .so files a binary needs and copies them to a directory. Since I found out that linuxdeploy doesn't do this already, at least not with every dependency.

Yeah there is a list of libs which don't get copied in linuxdeploy as those are expected to be in every linux distro.

However, go-appimage has a deploy everything mode that bypasses the list and bundles everything.

xplshn commented 2 months ago

Thanks for explaining everything. What about Schemas? 2024-07-07-124031_1366x768_scrot

It should work in a normal Linux system, however, this was generated in Musl, I am trying to bundle the LD.so so that it also works in Glibc, but I am not sure it will work. pelf_appdir_and_a_demo.tar.gz

Samueru-sama commented 2 months ago

linuxdeploy gtk does it: https://github.com/linuxdeploy/linuxdeploy-plugin-gtk/blob/master/linuxdeploy-plugin-gtk.sh#L247

xplshn commented 2 months ago

linuxdeploy gtk does it: https://github.com/linuxdeploy/linuxdeploy-plugin-gtk/blob/master/linuxdeploy-plugin-gtk.sh#L247

I generated my appdir with --plugin gtk

xplshn commented 2 months ago

I corrected something in the loader script, if you wish to try it, use this new archive. pelf_appdir_and_a_demo.tar.gz

I generated the bundle/blob with: "./pelf_appdir --output-to ./caja.blob --add-appdir ./caja.AppDir caja --add-ld-and-li bc /lib/ld-musl-x86_64.so.1 /lib/libc.musl-x86_64.so.1"

Check out the bundle's help page with --pbundle_help. Thanks for your help!

Samueru-sama commented 2 months ago

linuxdeploy gtk does it: https://github.com/linuxdeploy/linuxdeploy-plugin-gtk/blob/master/linuxdeploy-plugin-gtk.sh#L247

I generated my appdir with --plugin gtk

linuxdeploy gtk does it: https://github.com/linuxdeploy/linuxdeploy-plugin-gtk/blob/master/linuxdeploy-plugin-gtk.sh#L247

I generated my appdir with --plugin gtk

I can't see the GSETTINGS_SCHEMA_DIR being defined anywhere though. iirc linuxdeploy-gtk makes a few separate scripts from AppRun where that env variable is set.

edit: I haven't checked the new demo wait

edit2: yeah I can't see GSETTINGS_SCHEMA_DIR in the .blob and also the AppDir is lacking the usr/share/glib-2.0/schemas dir.

xplshn commented 2 months ago

Oh, I just read that part. Is there any way to make the linuxdeploy plugin create the AppRun? I think making PELF rely ONLY on AppDirs will be the better approach, having to do everything twice makes the code ugly and the loader slower...

xplshn commented 2 months ago

edit2: yeah I can't see GSETTINGS_SCHEMA_DIR in the .blob and also the AppDir is lacking the usr/share/glib-2.0/schemas dir.

I will create a new AppDir and try again once I make PELF handle the GSETTINGS_SCHEMA_DIR var

Samueru-sama commented 2 months ago

edit2: yeah I can't see GSETTINGS_SCHEMA_DIR in the .blob and also the AppDir is lacking the usr/share/glib-2.0/schemas dir.

I will create a new AppDir and try again once I make PELF handle the GSETTINGS_SCHEMA_DIR var

linuxdeploy-gtk should have set all of that.

If you want a reference, look inside the cpu-x appimage as that one uses said tool (in fact linuxdeploy-gtk was made by the same dev of cpu-x lol)

xplshn commented 2 months ago

I need a working AppDir, so that I can test the changes I make to PELF. Things made with Glibc in mind won't work in my setup.

Samueru-sama commented 2 months ago

I need a working AppDir, so that I can test the changes I make to PELF. Things made with Glibc in mind won't work in my setup.

Good luck finding a musl gtk appimage.

The AppRun in cpu-x seems to be have been fully autogenerated though.

#! /usr/bin/env bash

# autogenerated by linuxdeploy

# make sure errors in sourced scripts will cause this script to stop
set -e

this_dir="$(readlink -f "$(dirname "$0")")"

source "$this_dir"/apprun-hooks/"linuxdeploy-plugin-gettext.sh"
source "$this_dir"/apprun-hooks/"linuxdeploy-plugin-gtk.sh"
source "$this_dir"/apprun-hooks/"linuxdeploy-plugin-ncurses.sh"

exec "$this_dir"/AppRun.wrapped "$@"

AppRun.wrapped is a symlink to the binary

And each script inside apprun-hooks set a bunch of env variables, including export GSETTINGS_SCHEMA_DIR="$APPDIR//usr/share/glib-2.0/schemas"

xplshn commented 2 months ago

I think I figured out what I should do:

xplshn commented 2 months ago

If you have any AppDirs laying around that contain an AppRun, try the new release. I just pushed the changes.