Failed to build for apple darwin targets jemalloc-sys

[cavivie]

Try to build jemalloc-sys (0.5.4, 0.3.2), always get a ld linker error:

/usr/bin/ld: unrecognized option '-dynamic'

Follow issues:

• https://github.com/tpoechtrager/osxcross/issues/334
• https://github.com/tpoechtrager/cctools-port/issues/82

  Make sure /usr/local/osxcross/bin is in PATH.

Our cross image already sets osxcross path (/opt/osxcross/bin) in PATH.

Furthermore, we can find that the tools generated under the osxcross bin path all have the darwin version suffix (such as x86_64-apple-darwin20.04-clang), which will cause clang to be unable to find the correct ld (x86_64-apple-darwin-clang) when linking, and fallback to the default ld (/usr/bin/ld). Why generated targets with darwin version suffix, becasue osxcross build.sh does that by default:

• https://github.com/tpoechtrager/osxcross/blob/master/build.sh#L31-L58
• https://github.com/tpoechtrager/osxcross/blob/master/build.sh#L136

How to solve these problems: First we need to remove the darwin version suffix from the tool chain of the compiled target to make some cctools work correctly, so the most intuitive way we use is to generate new cctool symlinks without darwin version for cctools. Then we need put symlink in bin path such /usr/bin, if we won't do that, we need to ensure that osxcross bin path takes precedence over other bin path which containing ld searches. Since cross put the osxcross bin path at the end of PATH, we can put symlinks in /usr/bin, which will not conflict with other bin toolchains. Fixed by:

# file darwin.sh
create_arch_symlinks()
{
  local arch=$1 # support target arch
  local src_path=$2 # cctools bin source path
  local dst_path=$3 # cctools bin destination path
  local cctools=($(find $src_path -name "$arch-apple-darwin*-*"))
  local src_cctool
  local dst_cctool
  for src_cctool in ${cctools[@]}; do
    dst_cctool=$(echo "$src_cctool" | sed "s/$src_path/$dst_path/g")
    dst_cctool=$(echo "$dst_cctool" | sed -E "s/-darwin(.*)-/-darwin-/g")
    ln -sf $src_cctool $dst_cctool
  done
}

main() {
    local target_cpu="${1}"
    ...

    create_arch_symlinks $target_cpu /opt/osxcross/bin /usr/bin

    purge_packages
    ...
}

# file darwin-entry.sh
...
# should use first ar when existing mutilpe ar bins (head -n 1)
version=$(echo "${tools}" | grep 'ar$' | head -n 1 |  sed 's/'"${CROSS_TARGET}"'//' | sed 's/-ar//')
...

# file such as Docker.aarch64-apple-darwin-cross
...
ARG TARGET_CPU=aarch64
...
RUN /darwin.sh $TARGET_CPU
...

[Emilgardis]

I agree with all of this, however there is a much easier solution I think! wouldn't it be enough to just set CC_<target> to the correct path?

[cavivie]

wouldn't it be enough to just set CC_<target> to the correct path?

No, in the current example, the current problem is that CC_ has been set, not because of this setting, but because after the installed cctools carries the version suffix, the osxcross tool clang wrapper cannot correctly find its own linker (possibly other There are also more problems with c/c++ code compilation, at least jemalloc/ring cannot be compiled).

Note: the problem is that we build non native rust project (non pure rust project), so c/c++ compilation needs a cross-compile linker, but osxcross wrapper does not use the correct cross-compile linker.

[cavivie]

Well, it seems that things may not be that simple, and such a fix will cause osxcross to be unable to find the sdk based on the darwin version of the tool itself.

[cavivie]

Well, --target=arm64-apple-darwin does not seem to match the prefix of cctool=aarch64-apple-darwin20.4-clang.

configure:3067: aarch64-apple-darwin20.4-clang -O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-2 -fno-omit-frame-pointer --target=arm64-apple-darwin -Wall -O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-2 -fno-omit-frame-pointer --target=arm64-apple-darwin -Wall -O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-2 -fno-omit-frame-pointer --target=arm64-apple-darwin -Wall conftest.c  >&5
/usr/bin/ld: unrecognized option '-dynamic'
/usr/bin/ld: use the --help option for usage information

[cavivie]

After create symlink for x86_64-apple-darwin-clang target, we can build it.

After create symlink for aarch64-apple-darwin-clang target, we can not build it, because cc-rs generate a target arm64-apple-darwin for the osxcross clang wrapper, which does not match the prefix ot the osxcross clang wrapper, then we change the arm64 to aarch64, it works. But according to the issue: https://github.com/rust-lang/cc-rs/issues/542, target arm64-apple-darwin is the correct clang flag. So we do not change the arm64 to aarch64, create symlink for arm64-apple-darwin-clang target, and change the clang wrapper to arm64-apple-darwin20.4-clang, it also works.

In conclusion: The x86_64 apple darwin target compilation is different from the aarch64(arm64) apple darwin target. The clang wrapper aarch64-apple-darwin20.4-clang works with the target flag aarch64-apple-darwin. The clang wrapper arm64-apple-darwin20.4-clang works with the target flag arm64-apple-darwin. The clang wrapper aarch64-apple-darwin20.4-clang does not work with the target flag arm64-apple-darwin. The clang wrapper arm64-apple-darwin20.4-clang does not work with the target flag aarch64-apple-darwin.

UPD: when we use the target flag carries darwin version, we do not need to create symlinks for the clang wrapper. The clang wrapper aarch64-apple-darwin20.4-clang works with the target flag aarch64-apple-darwin20.4. The clang wrapper arm64-apple-darwin20.4-clang works with the target flag arm64-apple-darwin20.4. The clang wrapper aarch64-apple-darwin20.4-clang does not work with the target flag arm64-apple-darwin20.4. The clang wrapper arm64-apple-darwin20.4-clang does not work with the target flag aarch64-apple-darwin20.4.

[cavivie]

UPD: /opt/osxcross/bin contains both aarch64/arm64-apple-darwin clang, so the final root cause is that clang wrapper uses wrong ld (/usr/bin/ld), should use ld (/opt/osxcross/bin/xxx-ld).

Without modifying cross-rs, a simple solution like this worked for me:

# Cross.toml

[target.x86_64-apple-darwin]
image = "your image"
#pre-build = ["rm /usr/bin/ld && ln -s /opt/osxcross/bin/x86_64-apple-darwin20.4-ld /usr/bin/ld"]
env.passthrough = ["CFLAGS_x86_64_apple_darwin=-fuse-ld=x86_64-apple-darwin20.4-ld"]

[target.aarch64-apple-darwin]
image = "your image"
#pre-build = ["rm /usr/bin/ld && ln -s /opt/osxcross/bin/aarch64-apple-darwin20.4-ld /usr/bin/ld"]
env.passthrough = ["CFLAGS_aarch64_apple_darwin=-fuse-ld=aarch64-apple-darwin20.4-ld"]