Make AppImageKit more cross-compile friendly

[nurupo]

I want to cross-compile my program to several architectures (e.g. i386, x86_64, armel, armhf, etc.) and publish each as an AppImage. Besides cross-compiling my own application, I found that AppImageKit itself needs to be cross-compiled for each architecture in order to create the AppImage bundles for them. However, it has proven to be rather challenging to cross-compile AppImageKit.

src/build-runtime.sh.in and cmake/dependencies.cmake are the biggest cross-compilation offenders here.

• src/build-runtime.sh.in assumes that the build is made using system's native toolchain, it uses gcc, strip, readelf, ld, etc. without any way to specify the triplet prefix.
• cmake/dependencies.cmake builds dependencies not found on the system without ability to set --host= configure's argument, which results in the build failure when cross-compiling.

Here are full examples of how I have patched AppImageKit when cross-compiling it to armel and i386 on an x86_64 machine, so that you could see what I'm talking about and be able to reproduce:

armel

sudo docker run --rm -it debian:stretch-slim /bin/bash

dpkg --add-architecture armel
apt-get update
apt-get install -y --no-install-recommends ca-certificates libgtest-dev:armel git ca-certificates make libarchive-dev:armel automake autoconf libtool make gcc g++ libfuse-dev:armel liblzma-dev:armel libglib2.0-dev:armel libssl-dev:armel libinotifytools0-dev:armel liblz4-dev:armel libcairo-dev:armel desktop-file-utils cmake patch wget xxd patchelf

apt-get install -y --no-install-recommends binutils-arm-linux-gnueabi gcc-arm-linux-gnueabi g++-arm-linux-gnueabi

echo "
    SET(CMAKE_SYSTEM_NAME Linux)

    SET(CMAKE_C_COMPILER   /usr/bin/arm-linux-gnueabi-gcc)
    SET(CMAKE_CXX_COMPILER /usr/bin/arm-linux-gnueabi-g++)
    SET(CMAKE_AR           /usr/bin/arm-linux-gnueabi-gcc-ar CACHE FILEPATH "Archiver")

    SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
    SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
    SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
" > /usr/local/share/toolchain.cmake

cd /usr/src/gtest
cmake -DCMAKE_TOOLCHAIN_FILE=/usr/local/share/toolchain.cmake .
make install
cd -

# Make AppImageKit pick up liblzma.a instead of liblzma.so by hiding liblzma.so
mv /usr/lib/arm-linux-gnueabi/liblzma.so /usr/lib/arm-linux-gnueabi/foo-liblzma.so

git clone --recursive https://github.com/AppImage/AppImageKit
cd AppImageKit
git checkout c0ff6f738a65bfa890d5286d81d97754167237ea
# The broken stuff
sed -i 's|<SOURCE_DIR>/configure |<SOURCE_DIR>/configure --host=arm-linux-gnueabi |' cmake/dependencies.cmake
sed -i 's|gcc|/usr/bin/arm-linux-gnueabi-gcc|' src/build-runtime.sh.in
sed -i 's|ld |/usr/bin/arm-linux-gnueabi-ld |' src/build-runtime.sh.in
sed -i 's|objcopy |/usr/bin/arm-linux-gnueabi-objcopy |' src/build-runtime.sh.in
sed -i 's|readelf |/usr/bin/arm-linux-gnueabi-readelf |' src/build-runtime.sh.in
sed -i 's|strip|/usr/bin/arm-linux-gnueabi-strip|' src/build-runtime.sh.in
sed -i 's|objdump |/usr/bin/arm-linux-gnueabi-objdump |' src/build-runtime.sh.in
mkdir build
cd build
# Don't mind the pkg-config command, this is not broken, that's the proper way to do it
export PKG_CONFIG_LIBDIR=/usr/lib/arm-linux-gnueabi/pkgconfig:/usr/share/pkgconfig
cmake -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_TESTING=OFF -DCMAKE_TOOLCHAIN_FILE=/usr/local/share/toolchain.cmake -DUSE_SYSTEM_XZ=ON -DUSE_SYSTEM_INOTIFY_TOOLS=ON -DUSE_SYSTEM_LIBARCHIVE=ON -DUSE_SYSTEM_GTEST=ON ..
make VERBOSE=1
make install

# Restore it back
mv /usr/lib/arm-linux-gnueabi/foo-liblzma.so /usr/lib/arm-linux-gnueabi/liblzma.so

i386

sudo docker run --rm -it debian:stretch-slim /bin/bash

dpkg --add-architecture i386
apt-get update
apt-get install -y --no-install-recommends ca-certificates git lib32z1-dev libgtest-dev:i386 gcc-multilib g++-multilib libarchive-dev:i386 automake autoconf libtool make gcc g++ libfuse-dev:i386 liblzma-dev:i386 libglib2.0-dev:i386 libssl-dev:i386 libinotifytools0-dev:i386 liblz4-dev:i386 libcairo-dev:i386 desktop-file-utils cmake patch wget xxd patchelf

# There is no i386 cross-toolchain, you are supposed to use the multilib support for it, so let's create mock toolchain

echo '
#!/bin/sh
/usr/bin/x86_64-linux-gnu-gcc -m32 "$@"
' > /usr/bin/i386-linux-gnu-gcc
chmod +x /usr/bin/i386-linux-gnu-gcc

echo '
#!/bin/sh
/usr/bin/x86_64-linux-gnu-g++ -m32 "$@"
' > /usr/bin/i386-linux-gnu-g++
chmod +x /usr/bin/i386-linux-gnu-g++

echo '
#!/bin/sh
/usr/bin/x86_64-linux-gnu-ar "$@"
' > /usr/bin/i386-linux-gnu-ar
chmod +x /usr/bin/i386-linux-gnu-ar

echo "
    SET(CMAKE_SYSTEM_NAME Linux)

    SET(CMAKE_C_COMPILER   /usr/bin/i386-linux-gnu-gcc)
    SET(CMAKE_CXX_COMPILER /usr/bin/i386-linux-gnu-g++)
    SET(CMAKE_AR           /usr/bin/i386-linux-gnu-ar CACHE FILEPATH "Archiver")

    SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
    SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
    SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
" > /usr/local/share/toolchain.cmake

cd /usr/src/gtest
cmake -DCMAKE_TOOLCHAIN_FILE=/usr/local/share/toolchain.cmake .
make install
cd -

git clone --recursive https://github.com/AppImage/AppImageKit
cd AppImageKit
git checkout c0ff6f738a65bfa890d5286d81d97754167237ea
# The broken stuff
sed -i 's|gcc|/usr/bin/i386-linux-gnu-gcc|' src/build-runtime.sh.in
sed -i 's|ld |ld -m elf_i386 |' src/build-runtime.sh.in
mkdir build
cd build
# Don't mind the pkg-config command, this is not broken, that's the proper way to do it
export PKG_CONFIG_LIBDIR=/usr/lib/i386-linux-gnu/pkgconfig:/usr/share/pkgconfig
cmake -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_TESTING=OFF -DCMAKE_TOOLCHAIN_FILE=/usr/local/share/toolchain.cmake -DUSE_SYSTEM_XZ=ON -DUSE_SYSTEM_INOTIFY_TOOLS=ON -DUSE_SYSTEM_LIBARCHIVE=ON -DUSE_SYSTEM_GTEST=ON ..
make VERBOSE=1
make install

[TheAssassin]

You don't need to cross-compile appimagetool. You just need the right runtime for your platform. You can e.g., download the i386 runtime, and use it with appimagetool by passing via a CLI parameter.

[nurupo]

download the i386 runtime

Oh, I forgot to mention that I want to build AppImage from source, without downloading pre-built executables.

[TheAssassin]

I'm just trying to say that it should suffice cross-compile the runtime. That could save a lot of time.

[nurupo]

That's great to know. How do I cross-compile just the runtime?

[TheAssassin]

Well, you could just call make runtime, couldn't you? You'd still need to compile the dependencies, though.

Actually, I have plans to extract the runtime from the AppImageKit project, and move it into a separate repo. That'd make building it easier.

[nurupo]

Also, is the runtime enough for linuxdeployqt? Because I also use linuxdeployqt to -bundle-non-qt-libs -appimage my Qt program.

[TheAssassin]

linuxdeployqt bundles appimagetool, and it is hardcoded how appimagetool is called from it. Better don't use -appimage but download and call appimagetool directly with the correct options.

[probonopd]

Better don't use -appimage but download and call appimagetool directly with the correct options.

What do you mean by this?

[TheAssassin]

@probonopd please read the rest of the thread, and you'll see what is meant by that. There is no way to tell linuxdeployqt to tell appimagetool use a custom runtime with -appimage.

[probonopd]

Ah right, to use advanced options of linuxdeployqt one needs to run it manually, not using the -appimage option of linuxdeployqt. This is intentional so that only people who know what they are doing get exposed to the linuxdeployqt command line options.

[TheAssassin]

Ah right, to use advanced options of linuxdeployqt one needs to run it manually, not using the -appimage option of linuxdeployqt. This is intentional so that only people who know what they are doing get exposed to the linuxdeployqt command line options.

JFYI linuxdeploy's AppImage plugin aims to provide more configurability how the AppImage is generated.

[TheAssassin]

@nurupo I will try to rewrite the runtime build script in CMake today, that'll help making AppImageKit more cross compile friendly.

Perhaps we can even get rid of the 32-bit version of your AppImageBuild containers, build a multilib version of gcc and build the 32-bit versions of the dependencies in there as well. Then, the 32-bit builds could be created through cross compiling in the 64-bit container, using a toolchain like https://github.com/TheAssassin/linuxdeploy/blob/master/cmake/toolchains/i386-linux-gnu.cmake.

[nurupo]

Perhaps we can even get rid of the 32-bit version of your AppImageBuild containers

Um, those are not my containers at https://github.com/AppImage/AppImageBuild. My container is provided in the opening post of this thread and it does use multilib gcc to build 32-bit versions of my software and the AppImage on a 64-bit machine, notice the -m32 in my toolchain wrapper scripts, and it does use a CMake toolchain file to make CMake use those wrappers. Also, Debian's gcc has multilib support, there is no need to build gcc from source, just install gcc-multilib and g++-multilib packages.

[TheAssassin]

@nurupo the point is that we can make use of cross compilation support ourselves, and that increases the priority of this issue, which is in your interest.

I know you don't use AppImageBuild, but we do. And if you look into the Dockerfiles, you'll see that we don't use Debian, we use a really old version of CentOS. We're building our own compiler because it won't work with the outdated one the CentOS packages allow us to use. Same goes for a couple of other dependencies, e.g., GLib.

[nurupo]

Here is a bit more advanced script I use to build for amd64, arm64, armel, armhf, i386, mips, mips64el, mipsel, ppc64el and s390x architectures. It doesn't build successfully for all of them, qemu-user-static bugs out on some architectures, preventing things to be built. Don't remember for which exactly it bugs out, but it works just great for amd64, arm64, armel, armhf and i386. Note that the script is without a license for now. I'm planning on adding it later once I'm done doing what I'm doing, but if you want to use it now I could add it now.

[TheAssassin]

@nurupo thanks for the share, I'll check it out, but as I said the scripts are going to be "converted" to CMake code, which makes the whole build a bit more consistent and less complex. Then, a toolchain like I shared above should allow for cross-compiling everything without any major hassle.

[nurupo]

There is no need to convert those bash scripts to CMake code. Ideally you should just set the c compiler, linker, emulator and such in the toolchain file and it should just work with any toolchain, even android, for example.

As far as the AppImageKit goes, src/build-runtime.sh.in should use c compiler, linker, objcopy, readelf, etc. found by CMake (potentially from a toolchain file), e.g. CMAKE_C_COMPILER and such variables, instead of hardcoding gcc, ld and so on. It should also pass those c compiler and other CMake variables to any AppImageKit dependency the CMake compiles, e.g. mksquashfs, because just calling ./configure won't use the cross-compilation toolchain and will pick up the system gcc. It also needs to use CMAKE_CROSSCOMPILING_EMULATOR variable and CROSSCOMPILING_EMULATOR target property, which could be either empty or set to qemu by the user through a toolchain file when crosscompiling, to run programs that might potentially be of other architecture.

As far as LinuxDeployQt goes, I don't even know where to start. It hardcodes a lot of things and there is also this whole ldd issue. I made it work by overwriting the hardcoded programs it calls with scripts that call the programs I want it to use instead, but that's just a hack, it really needs a way to specify which programs to use instead of hardcoding their names. I believe you are also working on its replacement, so let's just not discuss LinuxDeployQt.

[TheAssassin]

818 replaced the build script with native CMake code. This allows for cross-compiling the runtime. Regarding the other dependencies, once #815 is resolved, we could think about going back to submodules and defining CMake targets ourselves instead of using the original build systems.

I tried to pass the right parameters to these build systems, but it didn't really work very well. Therefore I'd rather build the libraries using some custom CMake code in the future.

Regarding the ldd issue, linuxdeploy uses ldd, too, but only until we get to writing our own replacement (i.e., parse the ELF header field and implement our own search algorithm that works like ldd). See https://github.com/TheAssassin/linuxdeploy/blob/master/src/core/elf.cpp#L69. If you want to give it a try now, we can e.g., introduce an environment variable $LDD or something like that which can be used to override the path to the ldd that shall be used. Please feel free to try out linuxdeploy and open issues improving cross compiling support, your comments are highly appreciated.

[nurupo]

I see #818 still uses hardcoded objcopy. How about having a OBJCOPY_COMMAND variable which user can set, e.g. through a toolchain file, and if it's not set then it defaults to objcopy, and you actually check if the user-specified command or objcopy exist with find_program()?

Something like this

if (NOT OBJCOPY_COMMAND)
    set(OBJCOPY_COMMAND objcopy)
endif()
find_program(OBJCOPY_COMMAND_FOUND ${OBJCOPY_COMMAND})
if (NOT OBJCOPY_COMMAND_FOUND)
    message(FATAL_ERROR "Couldn't find \"${OBJCOPY_COMMAND}\" program. Make sure you have it installed. You can also specify path to it in OBJCOPY_COMMAND variable if it's still not being found.)
endif()
set(OBJCOPY_COMMAND ${OBJCOPY_COMMAND_FOUND})

# Now use ${OBJCOPY_COMMAND} in the commands

If user does

set(OBJCOPY_COMMAND /foo/bar/whatever-linux-gnu-objcopy)

in a toolchain file, then /foo/bar/whatever-linux-gnu-objcopy will be used. If the user doesn't set it, then objcopy will be used. CMake will try to confirm that both /foo/bar/whatever-linux-gnu-objcopy and objcopy programs actually exist.

[TheAssassin]

Any other programs which must be configurable?

[nurupo]

All of those. That's as of a01f60bb728af07a73644e677d73467985e8dca7 though, not sure if they are still used.

[TheAssassin]

Added a TOOLS_PREFIX in https://github.com/AppImage/AppImageKit/commit/f8697f5c84a885cc44f70a79e40fdb8cfa86c92f that you can set to make the search for programs prefer tools with that prefix, and use the non-prefixed version if there's none with the prefix. The paths are logged during configuration, making the whole process transparent to the user.

I will replace calls to the tools you listed here to use the programs found with check_program instead.

Then, that part should be fixed.

Regarding cross-compilation support, I'm making progress with the dependencies (our CMake infrastructure always supported cross-compilation). I'll push that stuff soon.

[TheAssassin]

https://github.com/AppImage/AppImageKit/tree/cross-compiling provides a way to cross compile AppImageKit from x86_64 to i386.

To test it, I'll try to cross-compile libappimage for AppImageLauncher using the provided toolchain file.

[TheAssassin]

32-bit cross compiling works fine now.

[nurupo]

Hm, now that I revisit my build script, I notice that I used qemu only for LinuxDeployQt, I don't use it for AppImageKit, so AppImageKit probably has no need to run foreign architecture programs, meaning that my suggestion of making AppImageKit use CMAKE_CROSSCOMPILING_EMULATOR is probably a pointless one.

[nurupo]

32-bit cross compiling works fine now.

What about armel?

[TheAssassin]

I did not try. But I didn't actually intend to close this issue (been a long week), I'll reopen this.

If you have concrete suggestions how we can improve cross-compiling support, please let me know.