search

aws / aws-lc-rs

aws-lc-rs is a cryptographic library using AWS-LC for its cryptographic operations. The library strives to be API-compatible with the popular Rust library named ring.
Other
311 stars 49 forks source link

Failure to build for windows using cross #514

Closed x528491x closed 2 months ago

x528491x commented 2 months ago

Problem:

I am trying to build my librespot fork for windows using cross, and my builds used to execute successfully in the past. I run into this error:

Relevant details

name = "aws-lc-rs" version = "1.8.1" dependencies = ["aws-lc-sys","mirai-annotations","paste","zeroize"]

name = "aws-lc-sys" version = "0.20.1" dependencies = ["bindgen","cc","cmake","dunce","fs_extra","libc","paste"]

System information:

Linux 6.10.7-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 30 00:08:59 UTC 2024 x86_64 unknown

Build log:

error: failed to run custom build command for `aws-lc-sys v0.20.1`

Caused by:
  process didn't exit successfully: `/target/release/build/aws-lc-sys-e1984f8bb5d8493f/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=AWS_LC_SYS_NO_PREFIX
  cargo:rerun-if-env-changed=AWS_LC_SYS_INTERNAL_BINDGEN
  cargo:rerun-if-env-changed=AWS_LC_SYS_EXTERNAL_BINDGEN
  cargo:rerun-if-env-changed=AWS_LC_SYS_NO_ASM
  cargo:rerun-if-env-changed=AWS_LC_SYS_CMAKE_BUILDER
  cargo:rerun-if-env-changed=AWS_LC_SYS_STATIC
  cargo:rerun-if-env-changed=CMAKE

  --- stderr
  Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html
  Missing dependency: nasm
  thread 'main' panicked at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/builder/main.rs:302:40:
  called `Result::unwrap()` on an `Err` value: "Required build dependency is missing. Halting build."
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
x528491x commented 2 months ago

The sudden dependency of this project arose for me because the newer version of hyper-rustls requires aws-lc.

Sricor commented 2 months ago

Same issue, x86_64-pc-windows-gnu compilation failed

error: failed to run custom build command for `aws-lc-sys v0.20.1`

Caused by:
  process didn't exit successfully: `D:\a\target\release\build\aws-lc-sys-761345a5fb95e42b\build-script-main` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=AWS_LC_SYS_NO_PREFIX
  cargo:rerun-if-env-changed=AWS_LC_SYS_INTERNAL_BINDGEN
  cargo:rerun-if-env-changed=AWS_LC_SYS_EXTERNAL_BINDGEN
  cargo:rerun-if-env-changed=AWS_LC_SYS_NO_ASM
  cargo:rerun-if-env-changed=AWS_LC_SYS_CMAKE_BUILDER
  cargo:rerun-if-env-changed=AWS_LC_SYS_STATIC
  cargo:rerun-if-env-changed=CMAKE

  --- stderr
  Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html
  Missing dependency: nasm
  thread 'main' panicked at C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\aws-lc-sys-0.20.1\builder/main.rs:302:40:
  called `Result::unwrap()` on an `Err` value: "Required build dependency is missing. Halting build."
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
justsmth commented 2 months ago

The issue is that NASM is not installed on the image.

Missing dependency: nasm

With the aws-lc-rs v1.9.0 release (hopefully later today?), you may choose to utilize prebuilt NASM binaries provided by our crate by setting the variable AWS_LC_SYS_PREBUILT_NASM=1 in the environment.

justsmth commented 2 months ago

As a temporary workaround, you could consider setting AWS_LC_SYS_NO_ASM=1 in the environment (and use a "debug" build). We have more information about our Windows build in our User Guide (see here and here). We will have updates to our User Guide when we release v1.9.0.

I'm sorry about the trouble here -- our Windows build has been painful for many users. We hope that our v1.9.0 release makes this much easier.

justsmth commented 2 months ago

The aws-lc-rs v1.9.0 release is now published. The new release provides a couple of ways to utilize prebuilt NASM objects if you prefer that option. We have more information about this in our (now updated) User Guide.

Please let us know if you have any problems with this! I'll leave this issue open for a few days to provide an opportunity for follow-up.

milen-denev commented 2 months ago

Hello, I have an issue exactly by updating to 1.9.0. Tried AWS_LC_SYS_PREBUILT_NASM="1" but didn't help. Installed NASM (winget install NASM) also didn't work. The way I tried to compile is by using cross and trying to cross compile from Windows to Linux. By reverting back to 1.8.1 worked as expected (cargo add aws-lc-rs@=1.8.1).

justinwsmith commented 2 months ago

By default, cross does not pass environment variables into the container. You might need to specify the variable in a Cross.toml like this: https://github.com/aws/aws-lc-rs/blob/main/Cross.toml#L24

You can see just above that how we get NASM installed: https://github.com/aws/aws-lc-rs/blob/main/Cross.toml#L11-L21

milen-denev commented 2 months ago

Thank you for the help, but I might be inexperienced in this area, still I wasn't able to compile it. My config code is:

Cross.toml

[target.x86_64-pc-windows-gnu]
pre-build = [
    "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable",
    ". $HOME/.cargo/env",
    "cargo install --force --locked bindgen-cli && mv $HOME/.cargo/bin/bindgen /usr/bin",
    "rm -rf $HOME/.cargo",
    "apt-get update && apt-get install --assume-yes nasm wget",
    "mkdir -pm755 /etc/apt/keyrings",
    "wget -NP /etc/apt/sources.list.d/ https://dl.winehq.org/wine-builds/ubuntu/dists/jammy/winehq-jammy.sources",
    "apt install --assume-yes --install-recommends winehq-stable",
]

[build.env]
passthrough = [
    "AWS_LC_SYS_PREBUILT_NASM"
]

./cargo/config.toml

[build]
# not related to this use case
rustflags = ["-C", "target-feature=+sse2"]

[env]
AWS_LC_SYS_PREBUILT_NASM="1"

And finally, part of 2 miles long error logs: 

exit status: 1
  cargo:warning=ToolExecError: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-std=c11" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/generated-include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/third_party/s2n-bignum/include" "-Wall" "-Wextra" "-Wno-unused-parameter" "-lpthread" "-ffile-prefix-map=/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0=" "-D_XOPEN_SOURCE=700" "-DBORINGSSL_IMPLEMENTATION=1" "-DBORINGSSL_PREFIX=aws_lc_0_21_0" "-o" "/target/x86_64-unknown-linux-gnu/release/build/aws-lc-sys-f182f22119339785/out/684db4233ee307ad-bignum_sub_p521.o" "-c" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/third_party/s2n-bignum/x86_att/p521/bignum_sub_p521.S" with args cc did not execute successfully (status code exit status: 1).cargo:warning=cc: error: unrecognized command line option '-ffile-prefix-map=/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0='

  exit status: 1
  cargo:warning=ToolExecError: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-std=c11" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/generated-include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/third_party/s2n-bignum/include" "-Wall" "-Wextra" "-Wno-unused-parameter" "-lpthread" "-ffile-prefix-map=/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0=" "-D_XOPEN_SOURCE=700" "-DBORINGSSL_IMPLEMENTATION=1" "-DBORINGSSL_PREFIX=aws_lc_0_21_0" "-o" "/target/x86_64-unknown-linux-gnu/release/build/aws-lc-sys-f182f22119339785/out/684db4233ee307ad-bignum_tolebytes_p521.o" "-c" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/third_party/s2n-bignum/x86_att/p521/bignum_tolebytes_p521.S" with args cc did not execute successfully (status code exit status: 1).

  --- stderr
  Evaluating: AWS_LC_SYS_PREBUILT_NASM='1'
  Parsed: AWS_LC_SYS_PREBUILT_NASM=true

  error occurred: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-std=c11" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/generated-include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/include" "-I" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/third_party/s2n-bignum/include" "-Wall" "-Wextra" "-Wno-unused-parameter" "-lpthread" "-ffile-prefix-map=/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0=" "-D_XOPEN_SOURCE=700" "-DBORINGSSL_IMPLEMENTATION=1" "-DBORINGSSL_PREFIX=aws_lc_0_21_0" "-o" "/target/x86_64-unknown-linux-gnu/release/build/aws-lc-sys-f182f22119339785/out/684db4233ee307ad-bignum_tolebytes_p521.o" "-c" "/cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.21.0/aws-lc/third_party/s2n-bignum/x86_att/p521/bignum_tolebytes_p521.S" with args cc did not execute successfully (status code exit status: 1).
justinwsmith commented 2 months ago

Ok, I think I see what's going on.

As a temporary workaround, you could possibly change the opt-level = 2 for release build in your Cargo.toml? There's Rust documentation about how to do this here.

[profile.release]
opt-level = 2

Sorry about this failure. I'm not sure why it's not showing up in our CI. We'll need to put out a patch update to address this.

Do you see anywhere in the output which compiler & version is being used? I'll see if I can to reproduce this.

justinwsmith commented 2 months ago

I was able to reproduce this failure by installing cross v0.2.5:

$ cargo install cross --locked --version 0.2.5
...
$ cross build --release --target x86_64-unknown-linux-gnu
...
.. .cargo:warning=cc: error: unrecognized command line option '-ffile-prefix-map=/project/aws-lc-sys='
...

When upgrading to the latest cross, we instead find that the compiler there has a critical bug:

$ cargo install cross --force  --git https://github.com/cross-rs/cross
...
$ cross build --release --target x86_64-unknown-linux-gnu
...
  --- stderr
  thread 'main' panicked at aws-lc-sys/builder/cc_builder.rs:271:13:
  ### COMPILER BUG DETECTED ###
  Your compiler (cc) is not supported due to a memcmp related bug reported in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95189. We strongly recommend against using this compiler.
...

The good news is that we have a way to reproduce the -ffile-prefix-map= build failure, so we'll be able to verify a fix.

milen-denev commented 2 months ago

@justinwsmith I want to thank you for your quick responses and acknowledgements and ensuring that it will be fixed.

About the question I am using Rust 1.81.0 version if this is what you are asking but I think this issue was also present in 1.80.1 before updating as well.

justsmth commented 2 months ago

With yesterday's release of aws-lc-sys v0.21.1, I think the issues listed here have been addressed:

Thanks for contact us about these issues! Feel free to let us know about any other questions/comments/concerns you may have.