Cross-compilation and multiarch

[vanfanel]

Hi there,

I was using an old Raspbian sysroot (living in /opt/rpi_root on my building X86 system) which had it's libs, headers and .so LD scripts in "standard" routes like /usr/lib, /lib, /usr/include, etc... Now (probably time ago, I just hadn't updated my Rpi crosscompilation sysroot) , the architecture-specific headers, libs and .so LD scripts have been moved their arm-linux-gnueabihf subdirs: that is, Raspbian has adopted the "multiarch" feature which allows the same lib to be present for different architectures on the same rootfs. The problem here is that the cross-compiler in this repository is not build with the "--enable-mutiarch --target=arm-linux-gnueabihf" configure options, so it looks for libs, crt*.o files, headers and .so LD scrips in "standard" pre-multiarch locations like /usr/lib, /lib, /us/include... without the arm-linux-gnueabihf subdirs. So, unless I am missing something here, I just don't understand how people can cross-compile these days. I have also tried building my own cross-compiler with crosstool-ng, but I can't see where it has support for "multiarch". What am I missing here? Thanks

[popcornmix]

I'm not sure you can generally use installed libs from raspbian when cross compiling. The problem is the dependencies have absolute paths which link back to you host's libraries. E.g.

ldd /usr/lib/arm-linux-gnueabihf/libssh.so
    linux-vdso.so.1 (0x7ef23000)
    librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76e9a000)
    libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0 (0x76d2c000)
    libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x76d05000)
    libgssapi_krb5.so.2 => /usr/lib/arm-linux-gnueabihf/libgssapi_krb5.so.2 (0x76cbc000)
    libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76b7f000)
    /lib/ld-linux-armhf.so.3 (0x54b15000)
    libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76b57000)
    libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x76b43000)
    libkrb5.so.3 => /usr/lib/arm-linux-gnueabihf/libkrb5.so.3 (0x76a8f000)
    libk5crypto.so.3 => /usr/lib/arm-linux-gnueabihf/libk5crypto.so.3 (0x76a51000)
    libcom_err.so.2 => /lib/arm-linux-gnueabihf/libcom_err.so.2 (0x76a3e000)
    libkrb5support.so.0 => /usr/lib/arm-linux-gnueabihf/libkrb5support.so.0 (0x76a25000)
    libkeyutils.so.1 => /lib/arm-linux-gnueabihf/libkeyutils.so.1 (0x76a19000)
    libresolv.so.2 => /lib/arm-linux-gnueabihf/libresolv.so.2 (0x769f4000)

Trying to link with libssh will pull in /usr/lib/arm-linux-gnueabihf/libk5crypto.so.3 from the host filesystem which won't exist (and if it didn't use arm-linux-gnueabihf it would be even worse as it may find an x86 library).

There are schemes involving a chroot with arm libraries that may make it possible, but it's certainly not easy. I'm not aware of a crosstool-ng config option that makes this all easier, but if you find one, I'd like to hear.

[seniorlackey]

I think I might be running into the problem described by the OP:

If I use a very simple test case (code available at the bottom) and I cross-compile it as follows:

bcm2708hardfp-cc -march=armv6 -mfpu=vfp -mfloat-abi=hard -o tst tst.c -lpthread

Now I log onto a board that has both of these:

$ ls -1 /lib/arm-linux-gnueabi*/libpthread.so.0 /lib/arm-linux-gnueabi/libpthread.so.0 /lib/arm-linux-gnueabihf/libpthread.so.0

ldd tells me that the hf/libpthread.so will be used as one would expect:

$ ldd ./tst libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6f75000) libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6e8f000) /lib/ld-linux.so.3 => /lib/ld-linux-armhf.so.3 (0xb6fa1000)

... but that does not appear to be true because:

$ LD_DEBUG=all ./tst ... 24421: trying file=/lib/arm-linux-gnueabi/libpthread.so.0 24421: 24421: file=libpthread.so.0 [0]; generating link map 24421: dynamic: 0xb6f25ef0 base: 0xb6f09000 size: 0x0001f214 24421: entry: 0xb6f0d544 phdr: 0xb6f09034 phnum: 9 ...

So it is not loading the 'hf' library, which would have been the correct behaviour.

This is probably not an issue on raspbian because only the 'hf' library is made available. But if both were made available like a regular Debian image, you would run into the same issue.

It does look like the toolchain (probably the linker, I imagine) is failing to set some bit that forces the 'hf' lib to be loaded.

This is the tst.c file:

#include <stdio.h>
#include <pthread.h>

static void * body(void * args)
{
        printf("Hello from %d\n", (int) pthread_self());
        return 0;
}

int main() {
        pthread_t me;

        if (0 == pthread_create(&me, NULL, body, NULL)) {
                printf("Thread launched...\n");
                pthread_join(me, NULL);
        }

        return 0;
}

[seniorlackey]

@popcornmix , Is it possible to build with "--enable-mutiarch" as suggested by the OP?

[amurzeau]

Moved comment to related issue #50 as it have more activity (https://github.com/raspberrypi/tools/issues/50#issuecomment-287636944)