Relative path to clang toolchain

[slonm]

When we use clang standalone toolchain not integrated in Linux when i may be installed in any path. I have made partial fix to allow place clang toolchain in any location and osxcross toolchain in some relative to clang location. The script called relative_rpath.sh should be places in the same location as build.sh and executed after build.sh

#!/usr/bin/env bash
#
# Fix ctool's and wrapper's rpath to relative.
#

find target/bin -type f | while read file; do
  RUNPATH=$(objdump -x "$file"  | grep RUNPATH | awk '{ print $2 }')
  if [ -z "$RUNPATH" ] || [[ "$RUNPATH" == '$ORIGIN/'* ]] ; then
    echo "$file already patched. Skipping"
  else
    RUNPATHR=$(realpath --relative-to="$(dirname $file)" "$RUNPATH")
    patchelf --set-rpath '$ORIGIN/'"$RUNPATHR" "$file"
    echo "Patched $file"
    objdump -x "$file"  | grep RUNPATH
  fi
done

It change all target binaries rpath to relative. That's works fine for me. Note: osxcross-conf exported paths left absolute and unchanged. Not sure is it uses somewhere

[robohack]

I've just encountered what I suspect is a similar problem, but with the osxcross binaries themselves (though perhaps that's what you mean by "clang toolchain").

Unfortunately I don't seem to have either the realpath or more importantly patchelf tools installed locally (I see patchelf appears to come from https://github.com/NixOS/patchelf, and realpath seems to be roughly the same as the slightly more common (on linux and some BSDs, but not macOS) readlink -f, though without --relative-to= -- in any case the script as-is won't work with realpath when RUNPATH is an actual multi-element path with colon separated strings, as it is on my most recent build).

In any case I don't think this (i.e. patchelf) is quite the best way to fix the problem. Ideally the $ORIGIN token should be injected into the rpath when these programs are initially linked.

There's one other library (on just x86_64-apple-darwin18-ld and xar that's too explicitly specified with an inflexible version number:

libcrypto.so.1.0.0 => not found

Ideally it should be linking with just libcrypto.so.1, but in no case should the final patch-level .0 be specified.

There's an old post by a Solaris developer talking about these issues, and the same ideas used for Solaris should work universally for all ELF platforms (though they have elfdump instead of objdump, and elfedit instead of patchelf); but clearly setting rpath at link time seems preferred, and I would strongly agree. https://blogs.oracle.com/solaris/avoiding-ldlibrarypath%3a-the-options-v2

(BTW, awk effectively as a built-in grep, so a separate invocation of grep is never needed when feeding a pipeline to awk. Just do awk '$1 == "RUNPATH" {...})

[tpoechtrager]

There's one other library (on just x86_64-apple-darwin18-ld and xar that's too explicitly specified with an inflexible version number:

libcrypto.so.1.0.0 => not found

Ideally it should be linking with just libcrypto.so.1, but in no case should the final patch-level .0 be specified.

It is the correct behavior. There's nothing to fix.

$ echo "int main(){}"| cc -xc - -o test -l crypto
$ ldd test
        linux-vdso.so.1 (0x00007ffcee9bf000)
        libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x00007fe389563000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fe3893a0000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fe38939b000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fe38937a000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fe389887000)

[robohack]

I guess I've been spoiled by the BSDs. The Linux systems I'm testing on do not have the shared library for libcrypto installed correctly, and clearly neither does yours.

The necessary symlinks for the libcrypto shared library are apparently not present, making programs linking with it depend on the exact version installed at build time, and breaking those programs when they are installed onto target systems which might have a slightly different patchlevel of the library installed.

Over on stackoverflow I see hints that on Linux it is necessary to have libssl-dev installed if you're going to be building things that rely on the libssl shared libraries such as libcrypto, though I've not yet discovered if this actually works for runtime dependency correction.

[robohack]

Hmmm... and then there's this: Linking to SSL shared library of different versions

So, let's ignore the libcrypto issue for the moment and focus on the issue of getting the RUNPATH to use "$ORIGIN".

[robohack]

Personally I would always choose static linking of all things, but especially things which are part of the same project, and probably also things with unstable and otherwise poor APIs such as OpenSSL.

[robohack]

So, I think I've solved almost all of the issues in making the osxcross tools somewhat more relocatable.

However what remains is that compiled programs requiring the the compiler runtime libraries end up with a runtime load path pointing to their original build location on the linux build host:

$ scp linux-build-host:src/thello ~/tmp/thello
$ ~/tmp/thello
dyld: Library not loaded: /home/gaw/work/osxcross/build/compiler-rt/build/lib/darwin/libclang_rt.ubsan_osx_dynamic.dylib
  Referenced from: /Users/gaw/tmp/thello
  Reason: image not found
Abort(coredump)
$ otool -L ~/tmp/thello       
/Users/gaw/tmp/thello:
        /home/gaw/work/osxcross/build/compiler-rt/build/lib/darwin/libclang_rt.ubsan_osx_dynamic.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
$ locate libclang_rt.ubsan_osx_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.ubsan_osx_dynamic.dylib
/Library/Developer/CommandLineTools/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.ubsan_osx_dynamic.dylib

This is how the test binary was built and linked:

$  x86_64-apple-darwin18-cc -v -g -fno-strict-aliasing -fstack-protector-all   -fsanitize=undefined -o thello thello.c
clang version 8.0.1 (tags/RELEASE_801/final)
Target: x86_64-apple-darwin18
Thread model: posix
InstalledDir: /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/bin
 "/home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/bin/clang-8" -cc1 -triple x86_64-apple-macosx10.9.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name thello.c -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -relaxed-aliasing -masm-verbose -munwind-tables -faligned-alloc-unavailable -target-sdk-version=10.14 -target-cpu core2 -dwarf-column-info -debug-info-kind=standalone -dwarf-version=2 -debugger-tuning=lldb -ggnu-pubnames -target-linker-version 409.12 -v -resource-dir /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1 -isystem /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/bin/../lib/clang/8.0.1/include -isysroot /home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk -cxx-isystem /home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk/usr/include/c++/v1 -Wno-liblto -fdebug-compilation-dir /home/gaw/src -ferror-limit 19 -fmessage-length 152 -fsanitize=alignment,array-bounds,bool,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,nonnull-attribute,null,pointer-overflow,return,returns-nonnull-attribute,shift-base,shift-exponent,signed-integer-overflow,unreachable,vla-bound,vptr -fsanitize-recover=alignment,array-bounds,bool,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,nonnull-attribute,null,pointer-overflow,returns-nonnull-attribute,shift-base,shift-exponent,signed-integer-overflow,vla-bound,vptr -stack-protector 3 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fobjc-runtime=macosx-10.9.0 -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/thello-35a0aa.o -x c thello.c
clang -cc1 version 8.0.1 based upon LLVM 8.0.1 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk/usr/local/include"
ignoring nonexistent directory "/home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk/Library/Frameworks"
ignoring duplicate directory "/home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/bin/../lib/clang/8.0.1/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/bin/../lib/clang/8.0.1/include
 /home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk/usr/include
 /home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk/System/Library/Frameworks (framework directory)
End of search list.
 "/home/gaw/opt/osxcross/bin/x86_64-apple-darwin18-ld" -demangle -lto_library /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -macosx_version_min 10.9.0 -syslibroot /home/gaw/opt/osxcross/bin/../SDK/MacOSX10.14.sdk -o thello /tmp/thello-35a0aa.o /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/lib/darwin/libclang_rt.ubsan_osx_dynamic.dylib -rpath @executable_path -rpath /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/lib/darwin -lSystem /home/gaw/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/lib/darwin/libclang_rt.osx.a
 "/home/gaw/opt/osxcross/bin/x86_64-apple-darwin18-dsymutil" -o thello.dSYM thello

Anyway, without the -fsanitize= option the binary runs A-OK on a test macOS client!

I'll follow up with another comment showing how I built and relocated the osxcross tools after they were built.

[robohack]

[[updated from first post with new arbitrary relocation support]]

It seems the osxcross tools can be relocated to any directory IFF they are built and linked with a library runtime path including ${ORIGIN}/../lib. The tricky part is getting the dollar sign passed through both make and libtool.

I.e. you can relocate the tools outside of the osxcross source directory to any install location you desire once they are built (though the issue with the compiler runtime library runtime path for cross-compiled macOS binaries, as described in my previous note, remains).

Here's how I did it:

• Install Clang/LLVM and CMake from public binary distributions (e.g. in $HOME/opt)

cd ~/tmp
curl -L -O https://github.com/llvm/llvm-project/releases/download/llvmorg-8.0.1/clang+llvm-8.0.1-x86_64-linux-sles11.3.tar.xz
cd ../opt
tar -xJf ../tmp/clang+llvm-8.0.1-x86_64-linux-sles11.3.tar.xz

PATH=$HOME/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/bin:$PATH

cd ~/tmp
curl -L -O https://github.com/Kitware/CMake/releases/download/v3.15.2/cmake-3.15.2-Linux-x86_64.tar.gz
cd ../opt
tar -xf ../tmp/cmake-3.15.2-Linux-x86_64.tar.gz

PATH=$HOME/opt/cmake-3.15.2-Linux-x86_64/bin:$PATH

• check out osxcross:

  cd ~/work
  git clone https://github.com/tpoechtrager/osxcross
  

• get the MacOSX SDK (run this over on a macOS machine, of course):

  cd ~/work
  git clone https://github.com/tpoechtrager/osxcross
  cd osxcross
  UNATTENDED=1 ./tools/gen_sdk_package.sh
  scp tarballs/MacOSX10.14.sdk.tar.xz linux-build-host:work/osxcross/tarballs/
  

• build and "install" osxcross (note the very careful quoting and escaping of the second -rpath parameter):

  cd ~/work/osxcross
  CC="clang" CXX="clang++ -stdlib=libc++" LDFLAGS="-L$HOME/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib -Wl,-rpath,$HOME/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib,-rpath,'\$\${ORIGIN}/../lib'" UNATTENDED=1 ./build.sh > build.out 2>&1
  mv target ~/opt/osxcross-MacOSX10.14-clang-8
  

• add osxcross to your PATH and away you go!

  PATH=$HOME/opt/osxcross-MacOSX10.14-clang-8/bin:$PATH
  

• also build the compiler runtime libraries (though as per the previous note, they won't be terribly useful until they get a proper runtime library path for any target macOS client):

  cd ~/work/osxcross
  UNATTENDED=1 ./build_compiler_rt.sh > build_compiler_rt.out 2>&1
  

• follow the instructions given at the end of build_compiler_rt.out, essentially:

  mkdir -p ~/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/include
  mkdir -p ~/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/lib/darwin
  cp -rv build/compiler-rt/include/sanitizer ~/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/include
  cp -v build/compiler-rt/build/lib/darwin/*.a ~/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/lib/darwin
  cp -v build/compiler-rt/build/lib/darwin/*.dylib ~/opt/clang+llvm-8.0.1-x86_64-linux-sles11.3/lib/clang/8.0.1/lib/darwin
  

[twodayslate]

If you install clang into target:

INSTALLPREFIX=`pwd`/target ./build_clang.sh

The build the toolchain you then just need to do:

PATH=`pwd`/target/bin:$PATH LDFLAGS="-L"`pwd`/target/lib" -Wl,-rpath,'\$\${ORIGIN}/../lib'" ./build.sh

Everything is then self contained in the target folder. You can alternatively just download clang from llvm-project and extract it to the target folder.

It would be great if this was in the README somewhere that paths aren't relative.