search

cross-rs / cross

“Zero setup” cross compilation and “cross testing” of Rust crates
Apache License 2.0
6.42k stars 359 forks source link

Cross build for x86_64-apple-darwin fails on openssl, ranlib malformed archive #1243

Open xrl opened 1 year ago

xrl commented 1 year ago

Checklist

Describe your issue

I am compiling a project with cross for the first time, I am targeting x86_64-apple-darwin and failing. The CI system runs on r5.2xlarge's in AWS, regular Intel servers. I have posted a full copy of the build failure in this gist: https://gist.github.com/xrl/208a57f721e673e8ace2666c16ae87d1 . And for more context, here are my project's configurations:

# Cargo.toml
[package]
name = "ents-cli"
version = "0.3.3"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[[bin]]
name = "ents"

[dependencies]
aws-arn = "0.2.0"
aws-config = "0.55.0"
aws-sdk-ecr = "0.25.1"
base64 = "0.13.0"
chrono = "0.4.19"
clap = "3.0.0-rc.0"
dirs-next = "2"
env_logger = "0"
eyre = "0"
futures-util = "0.3"
http = "0.2.4"
k8s-openapi = { version = "0.17.0", features = ["v1_25"] }
kube = { version="=0.80.0", default-features = false, features = ["client"] }
lazy_static = "1"
log = "0"
openssl = { version = "0.10", features = ["vendored"] }
phf = { version = "0.10.0", features = ["macros"] }
prettytable-rs = "0.8.0"
rancher = "0"
reqwest = "0"
semver = "1"
serde = {version = "1", features = ["derive"]}
serde_json = "1"
serde_yaml = "0.8.17"
tokio = { version = "1", features = ["full"]}
tokio-stream = { version = "0" }
which = "4"

and I figured out how to use build my own OS X SDKs (turns out I had an SDK kicking around from a rust-musl-builder attempt). I am using MacOSX12.0.sdk.tar.xz which I package for my own use. And I create my Darwin cross images in my CI with these steps:

steps:
  - name: build
    image: ubuntu:22.04
    pull: always
    resources:
      cpu: 3700m
      memory: 62000MiB
    environment:
      SDK_URL: https://my-sdk-hosting/macos-sdks/MacOSX12.0.sdk.tar.xz
    commands:
      - apt-get update > /dev/null; apt-get install -y docker.io curl > /dev/null
      - curl -O https://ids-dev-artifacts.s3.amazonaws.com/ents-cli/0.3.3/ents-x86_64-unknown-linux-musl; chmod a+x ents-*; mv ents-* /usr/local/bin/ents
      - ents ecr docker-login --stripe-group=ids --only=ids-dev
      - git submodule update --init --remote
      - cd docker/
      - docker build -t $REPO:x86_64-apple-darwin -f cross-toolchains/docker/Dockerfile.x86_64-apple-darwin-cross --build-arg "MACOS_SDK_URL=$SDK_URL" .
      - docker build -t $REPO:aarch64-apple-darwin -f cross-toolchains/docker/Dockerfile.aarch64-apple-darwin-cross --build-arg "MACOS_SDK_URL=$SDK_URL" .
      - docker push $REPO:x86_64-apple-darwin; docker push $REPO:aarch64-apple-darwin

those images are then referenced from my Rust project:

# Cross.toml
[target.aarch64-apple-darwin]
image = "539703851677.dkr.ecr.us-west-2.amazonaws.com/cross:aarch64-apple-darwin"

[target.x86_64-apple-darwin]
image = "539703851677.dkr.ecr.us-west-2.amazonaws.com/cross:x86_64-apple-darwin"

and finally my CI pipeline triggering the work:

# .drone.yml
---
kind: pipeline
type: kubernetes
name: default

metadata:
  namespace: drone
  labels:
    app: ents-cli
  annotations:
    iam.amazonaws.com/role: k8s-drone
service_account_name: drone-build-sa

environment:
  RUST_VERSION: 1.68.1
  DEBIAN_FRONTEND: noninteractive
  AWS_REGION: us-west-2
  CROSS_VERSION: 0.2.5
  DOCKER_HOST: tcp://localhost:2375

services:
  - name: docker
    image: docker:dind
    privileged: true
    entrypoint: ["dockerd", "--host=tcp://0.0.0.0:2375"] # work around for default TLS. I don't want to bother sharing the cert dir.

workspace:
  path: /root

steps:
  - name: cross-compile-all-targets
    image: ubuntu:22.04
    pull: always
    resources:
      cpu: 3700m
      memory: 62000MiB
    commands:
      - apt-get update > /dev/null; apt-get install -y curl build-essential docker.io time > /dev/null
      - curl -O https://ids-dev-artifacts.s3.amazonaws.com/ents-cli/0.3.3/ents-x86_64-unknown-linux-musl; chmod a+x ents-*; mv ents-* /usr/local/bin/ents
      - ents ecr docker-login --stripe-group=ids --only=ids-dev
      - time curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain=$RUST_VERSION --target x86_64-apple-darwin aarch64-apple-darwin -y
      - . "$HOME/.cargo/env"
      - cargo fmt --check
      - time curl -LO --silent https://github.com/cross-rs/cross/releases/download/v$CROSS_VERSION/cross-x86_64-unknown-linux-gnu.tar.gz; tar xzf cross-*.tar.gz; mkdir -p ~/.cargo/bin; mv cross ~/.cargo/bin
      - for TARGET in x86_64-unknown-linux-musl x86_64-apple-darwin aarch64-apple-darwin; do echo "==================\n===== WORKING ON $TARGET =====\n=================="; time cross build --target $TARGET --release; mv target/$TARGET/release/ents target/$TARGET/release/ents-$TARGET; done

What target(s) are you cross-compiling for?

other (specify in description)

Which operating system is the host (e.g computer cross is on) running?

What architecture is the host?

What container engine is cross using?

cross version

cross 0.4.2

Example

No response

Additional information / notes

No response

Emilgardis commented 1 year ago 
  --- stderr
  x86_64-apple-darwin21.1-ar: creating archive apps/libapps.a
  x86_64-apple-darwin21.1-ar: creating archive libcrypto.a
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(async_null.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(async_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(cms_cd.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(dso_dl.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(dso_dlfcn.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(dso_vms.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(dso_win32.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(ebcdic.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(ecp_nistp224.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(ecp_nistp256.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(ecp_nistp521.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(ecp_nistputil.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(e_camellia.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(e_idea.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(e_rc5.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(e_seed.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(m_md2.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(rand_egd.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(rand_vms.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(rand_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(threads_none.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libcrypto.a(threads_win.o) has no symbols
  x86_64-apple-darwin21.1-ar: creating archive libssl.a
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libssl.a(ssl_utst.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin21.1-ranlib: file: libssl.a(t1_trce.o) has no symbols
  ranlib: /target/x86_64-apple-darwin/release/build/openssl-sys-aa87580c099679b3/out/openssl-build/install/lib/libcrypto.a.new: malformed archive
  make: *** [Makefile:326: install_dev] Error 1
  thread 'main' panicked at '

  Error installing OpenSSL:
      Command: cd "/target/x86_64-apple-darwin/release/build/openssl-sys-aa87580c099679b3/out/openssl-build/build/src" && "make" "install_dev"
      Exit status: exit status: 2

      ', /cargo/registry/src/github.com-1ecc6299db9ec823/openssl-src-111.25.2+1.1.1t/src/lib.rs:505:13
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Command exited with non-zero status 101
28.06user 1.78system 4:06.25elapsed 12%CPU (0avgtext+0avgdata 160680maxresident)k
0inputs+709912outputs (0major+219717minor)pagefaults 0swaps

Not sure how to go forward with this

xrl commented 1 year ago

I will get a newer SDK and rebuild the cross images and try again.

Edit same failure mode, just a slight version bump in the ar being invoked.

anotherstarburst commented 1 year ago

I've started encountering this issue too in a fresh environment.

cargo build works as expected in both MacOS (M2) and when running in a linux Docker container. I also have

openssl = { version = "0.*", features = ["vendored"] }

I'm using the same *-apple-darwin targets I had used previously and that worked.

gh0st-work commented 9 months ago

Same issue

Here's more detailed example: GitHub Actions logs & config. Also openssl -> libcrypto.

x86_x64-apple-darwin log:

install libcrypto.a -> /target/x86_64-apple-darwin/release/build/openssl-sys-0d18ecb4af7331db/out/openssl-build/install/lib/libcrypto.a

  --- stderr
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libssl.a(libssl-lib-ssl_utst.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: providers/libdefault.a(libdefault-lib-rand_cpu_x86.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: providers/libdefault.a(libdefault-lib-rand_tsc.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: providers/libdefault.a(libdefault-lib-rand_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-async_null.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-async_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-cms_cd.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_dl.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_dlfcn.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_vms.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_win32.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-f_impl32.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-e_idea.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-e_rc5.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-ebcdic.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-threads_none.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-threads_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-async_null.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-async_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-cms_cd.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_dl.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_dlfcn.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_vms.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-dso_win32.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-f_impl32.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-e_idea.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-e_rc5.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-ebcdic.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-threads_none.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-threads_win.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libcrypto-lib-rand_deprecated.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libdefault-lib-rand_cpu_x86.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libdefault-lib-rand_tsc.o) has no symbols
  /opt/osxcross/bin/x86_64-apple-darwin20.4-ranlib: file: libcrypto.a(libdefault-lib-rand_win.o) has no symbols
  ranlib: /target/x86_64-apple-darwin/release/build/openssl-sys-0d18ecb4af7331db/out/openssl-build/install/lib/libcrypto.a.new: malformed archive
  make: *** [Makefile:2309: install_dev] Error 1
  thread 'main' panicked at /cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-src-300.1.6+3.1.4/src/lib.rs:585:9:

  Error installing OpenSSL:
      Command: cd "/target/x86_64-apple-darwin/release/build/openssl-sys-0d18ecb4af7331db/out/openssl-build/build/src" && "make" "install_dev"
      Exit status: exit status: 2

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
Error: Process completed with exit code 101.

aarch64-apple-darwin log:

 install libcrypto.a -> /target/aarch64-apple-darwin/release/build/openssl-sys-311a69fe80b28e78/out/openssl-build/install/lib/libcrypto.a

  --- stderr
  ranlib: /target/aarch64-apple-darwin/release/build/openssl-sys-311a69fe80b28e78/out/openssl-build/install/lib/libcrypto.a.new: malformed archive
  make: *** [Makefile:2297: install_dev] Error 1
  thread 'main' panicked at /cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-src-300.1.6+3.1.4/src/lib.rs:585:9:

  Error installing OpenSSL:
      Command: cd "/target/aarch64-apple-darwin/release/build/openssl-sys-311a69fe80b28e78/out/openssl-build/build/src" && "make" "install_dev"
      Exit status: exit status: 2

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
Error: Process completed with exit code 101.

Tried to change openssl vendored version, this probably cannot change anything as they fetch latest package version (I expect, as I would develop it in a such way).


Separate issue

There's also separate issue (in same GHA workflow). Encountering in *-pc-windows-msvc run:

#17 6.080 + apt-get update
#17 6.269 Get:1 https://dl.winehq.org/wine-builds/ubuntu bionic InRelease [6263 B]
#17 6.280 Hit:2 http://archive.archive.ubuntu.com/ubuntu focal InRelease
#17 6.293 Hit:3 http://archive.archive.ubuntu.com/ubuntu focal-updates InRelease
#17 6.306 Hit:4 http://archive.archive.ubuntu.com/ubuntu focal-backports InRelease
#17 6.319 Err:1 https://dl.winehq.org/wine-builds/ubuntu bionic InRelease
#17 6.319   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 76F1A20FF987672F
#17 6.357 Hit:5 http://ports.ubuntu.com/ubuntu-ports focal InRelease
#17 6.425 Hit:6 http://security.archive.ubuntu.com/ubuntu focal-security InRelease
#17 6.437 Hit:7 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
#17 6.516 Hit:8 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
#17 6.596 Hit:9 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
#17 6.994 Reading package lists...
#17 7.873 W: GPG error: https://dl.winehq.org/wine-builds/ubuntu bionic InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 76F1A20FF987672F
#17 7.873 E: The repository 'https://dl.winehq.org/wine-builds/ubuntu bionic InRelease' is not signed.
#17 ERROR: process "/bin/sh -c /wine.sh" did not complete successfully: exit code: 100
------
 > [11/17] RUN /wine.sh:
6.319 Err:1 https://dl.winehq.org/wine-builds/ubuntu bionic InRelease
6.319   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 76F1A20FF987672F
6.357 Hit:5 http://ports.ubuntu.com/ubuntu-ports focal InRelease
6.425 Hit:6 http://security.archive.ubuntu.com/ubuntu focal-security InRelease
6.437 Hit:7 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
6.516 Hit:8 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
6.596 Hit:9 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease

7.873 W: GPG error: https://dl.winehq.org/wine-builds/ubuntu bionic InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 76F1A20FF987672F
7.873 E: The repository 'https://dl.winehq.org/wine-builds/ubuntu bionic InRelease' is not signed.
------
Dockerfile.x86_64-pc-windows-msvc-cross:18
--------------------
  16 |     
  17 |     COPY wine.sh /
  18 | >>> RUN /wine.sh
  19 |     
  20 |     # need windows-style perl for paths
--------------------
ERROR: failed to solve: process "/bin/sh -c /wine.sh" did not complete successfully: exit code: 100
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c /wine.sh" did not complete successfully: exit code: 100

Also appears in another (not mine) project GHA logs. Not dependent on openssl. I would probably submit it, if no one will say where I'm an idiot or smth like that in a few days :) Not found anything in issues & PRs for now.

rnza0u commented 3 months ago

Same problem here.

Anyone got lucky on this one ?

EDIT :

Just found a fix.

For some reason, the appropriate ranlib executable that is in /opt/osxcross/bin/arm64e-apple-darwin21.4-ranlib is not the one being used during OpenSSL compilation.

I found that line in the logs:

running cd "/target/aarch64-apple-darwin/release/build/openssl-sys-17f6230b0d6ee208/out/openssl-build/build/src" && env -u CROSS_COMPILE AR="aarch64-apple-darwin21.4-ar" CC="aarch64-apple-darwin21.4-clang" RANLIB="ranlib" "perl" "./Configure" "--prefix=/target/aarch64-apple-darwin/release/build/openssl-sys-17f6230b0d6ee208/out/openssl-build/install" "--openssldir=/usr/local/ssl" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "darwin64-arm64-cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=arm64-apple-darwin" "-stdlib=libc++" "-fuse-ld=aarch64-apple-darwin21.4-ld"

As you can see, the RANLIB variable is set to ranlib, so it will actually execute the default ranlib executable in the current system, which does not understand OSX build archives.

I had to do the followings to fix the problem :

[build.env]
passthrough = [
    "RANLIB"
]