search

PyO3 / maturin

Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages
https://maturin.rs
Apache License 2.0
3.99k stars 275 forks source link

Strange behavior with PyPy in 0.10 prerelease #478

Closed messense closed 3 years ago

messense commented 3 years ago

Please provide the following information:

Please list the exact steps required to reproduce your error with all command output and if possible with a repository:

maturin-0.10.0b6 , https://github.com/milesgranger/pyrus-cramjam/runs/2218894957?check_suite_focus=true errored with

maturin build -i /opt/hostedtoolcache/PyPy/3.6.12/x64/bin/pypy --release --out dist
🔗 Found pyo3 bindings
🐍 Found PyPy 3.6 at /opt/hostedtoolcache/PyPy/3.6.12/x64/bin/pypy
📦 Built source distribution to dist/cramjam-2.1.0.tar.gz
warning: version requirement `0.6.0+zstd.1.4.8` for dependency `zstd` includes semver metadata which will be ignored, removing the metadata is recommended to avoid confusion
   Compiling cfg-if v1.0.0
   Compiling scopeguard v1.1.0
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-musl` target may not be installed

error: aborting due to previous error

error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-musl` target may not be installed

error: aborting due to previous error

error: could not compile `scopeguard`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `cargo rustc --message-format json --manifest-path Cargo.toml --release --target x86_64-unknown-linux-musl --lib --`

It's strange that --target x86_64-unknown-linux-musl is passed to cargo rustc, it works fine with maturin 0.9.4

messense commented 3 years ago 
Run make pypy-build
printenv
POWERSHELL_DISTRIBUTION_CHANNEL=GitHub-Actions-ubuntu20
ANDROID_NDK_ROOT=/usr/local/lib/android/sdk/ndk-bundle
GITHUB_EVENT_NAME=push
GITHUB_ACTION=run2
GITHUB_REPOSITORY=messense/pyrus-cramjam
GRADLE_HOME=/usr/share/gradle
GITHUB_WORKFLOW=CI
PERFLOG_LOCATION_SETTING=RUNNER_PERFLOG
JOURNAL_STREAM=9:20396
LEIN_HOME=/usr/local/lib/lein
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
GOROOT_1_16_X64=/opt/hostedtoolcache/go/1.16.2/x64
GITHUB_BASE_REF=
GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_1631e229-1032-45b1-8424-be397b8375f9
_=/usr/bin/make
ANDROID_NDK_LATEST_HOME=/usr/local/lib/android/sdk/ndk/22.0.7026061
RUNNER_PERFLOG=/home/runner/perflog
RUNNER_TOOL_CACHE=/opt/hostedtoolcache
DOTNET_MULTILEVEL_LOOKUP=0
DEPLOYMENT_BASEPATH=/opt/runner
DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
PATH=/opt/hostedtoolcache/PyPy/3.6.12/x64/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ImageOS=ubuntu20
GITHUB_ACTION_REF=
GRAALVM_11_ROOT=/usr/local/graalvm/graalvm-ce-java11-21.0.0.2
GITHUB_SERVER_URL=https://github.com
PIPX_BIN_DIR=/opt/pipx_bin
PIPX_HOME=/opt/pipx
GITHUB_RUN_NUMBER=25
INVOCATION_ID=b26d52d28bae473d9271824bdf42ae91
ANDROID_HOME=/usr/local/lib/android/sdk
GITHUB_ACTOR=messense
LEIN_JAR=/usr/local/lib/lein/self-installs/leiningen-2.9.5-standalone.jar
AZURE_EXTENSION_DIR=/opt/az/azcliextensions
CONDA=/usr/share/miniconda
JAVA_HOME_8_X64=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64
HOMEBREW_CELLAR="/home/linuxbrew/.linuxbrew/Cellar"
ANT_HOME=/usr/share/ant
ImageVersion=20210318.0
GITHUB_REPOSITORY_OWNER=messense
HOME=/home/runner
GECKOWEBDRIVER=/usr/local/share/gecko_driver
RUNNER_TEMP=/home/runner/work/_temp
VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg
GITHUB_REF=refs/heads/buffer-protocol
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_1631e229-1032-45b1-8424-be397b8375f9
ANDROID_NDK_HOME=/usr/local/lib/android/sdk/ndk-bundle
CI=true
pythonLocation=/opt/hostedtoolcache/PyPy/3.6.12/x64/bin
HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS=3650
GOROOT_1_15_X64=/opt/hostedtoolcache/go/1.15.10/x64
GITHUB_JOB=pypy
JAVA_HOME=/usr/lib/jvm/adoptopenjdk-11-hotspot-amd64
USER=runner
GITHUB_ACTION_REPOSITORY=
BOOTSTRAP_HASKELL_NONINTERACTIVE=1
GITHUB_ACTIONS=true
GITHUB_HEAD_REF=
DOTNET_NOLOGO=1
RUNNER_TRACKING_ID=github_118b9323-74cd-4865-b595-f68a70ada47e
MAKEFLAGS=
MFLAGS=
HOMEBREW_PREFIX="/home/linuxbrew/.linuxbrew"
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
HOMEBREW_REPOSITORY="/home/linuxbrew/.linuxbrew/Homebrew"
ACCEPT_EULA=Y
HOMEBREW_NO_AUTO_UPDATE=1
GOROOT_1_14_X64=/opt/hostedtoolcache/go/1.14.15/x64
RUNNER_USER=runner
GITHUB_API_URL=https://api.github.com
ANDROID_SDK_ROOT=/usr/local/lib/android/sdk
***
GITHUB_RUN_ID=701759626
SWIFT_PATH=/usr/share/swift/usr/bin
DEBIAN_FRONTEND=noninteractive
CHROME_BIN=/usr/bin/google-chrome
GITHUB_WORKSPACE=/home/runner/work/pyrus-cramjam/pyrus-cramjam
GITHUB_SHA=731a843c9d0506842470cd2b3e06f78410d6aa6b
RUNNER_WORKSPACE=/home/runner/work/pyrus-cramjam
CHROMEWEBDRIVER=/usr/local/share/chrome_driver
SHLVL=1
JAVA_HOME_11_X64=/usr/lib/jvm/adoptopenjdk-11-hotspot-amd64
RUNNER_OS=Linux
GITHUB_RETENTION_DAYS=90
SELENIUM_JAR_PATH=/usr/share/java/selenium-server-standalone.jar
LANG=C.UTF-8
MAKELEVEL=1

printenv shows no sign of CARGO_BUILD_TARGET.

konstin commented 3 years ago

Sounds like Platform::guess_current() guessed wrong. I think this can be fixed by reverting target: &str to target: Option<&str> in compile_target, because than we aren't reliant on the guess of platforms. If you can confirm that platforms is wrong for that ci environment it would also make sense to report this upstream.

messense commented 3 years ago

Sounds like Platform::guess_current() guessed wrong. I think this can be fixed by reverting target: &str to target: Option<&str> in compile_target, because than we aren't reliant on the guess of platforms. If you can confirm that platforms is wrong for that ci environment it would also make sense to report this upstream.

🔗 Found pyo3 bindings
Target: Target {
    os: Linux,
    arch: X86_64,
    env: Some(
        GNU,
    ),
    triple: "x86_64-unknown-linux-gnu",
}
🐍 Found PyPy 3.6 at /opt/hostedtoolcache/PyPy/3.6.12/x64/bin/pypy

I've tested it in https://github.com/messense/pyrus-cramjam/runs/2228798334?check_suite_focus=true , it seems fine. 😱

al626 commented 3 years ago

Also causing issues in CPython builds for rust-numpy: https://github.com/PyO3/rust-numpy/pull/179/checks?check_run_id=2260903749

poetry run maturin develop
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at python
   Compiling cfg-if v1.0.0
   Compiling smallvec v1.6.1
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-musl` target may not be installed

error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-musl` target may not be installed

error: aborting due to previous error

error: aborting due to previous error

error: could not compile `cfg-if`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `cargo rustc --message-format json --manifest-path Cargo.toml --target x86_64-unknown-linux-musl --lib --`
Error: Process completed with exit code 1.

Strangely, if I run the above with a locally built (debug) version of maturin (tag v0.10.0) it succeeds. I've instrumented this to print out version info and musl is not mentioned:

diff --git a/src/target.rs b/src/target.rs
index f875c44..4b6623e 100644
--- a/src/target.rs
+++ b/src/target.rs
@@ -107,6 +107,9 @@ impl Target {
             (Os::Windows, Arch::Armv7L) => bail!("armv7l is not supported for Windows"),
             (_, _) => {}
         }
+
+        println!("Created target: {:?} {:?}, {:?}: {:?}", os, arch, platform.target_env, platform.target_triple);
+
         Ok(Target {
             os,
             arch,
Created target: Linux X86_64, Some(GNU): "x86_64-unknown-linux-gnu"

Also works with v0.9.4 from pip.

al626 commented 3 years ago

Sounds like Platform::guess_current() guessed wrong. I think this can be fixed by reverting target: &str to target: Option<&str> in compile_target, because than we aren't reliant on the guess of platforms. If you can confirm that platforms is wrong for that ci environment it would also make sense to report this upstream.

Yep, platform::guess_current looks like it works based on the env (platform is built on)[https://docs.rs/platforms/0.2.1/src/platforms/target/env.rs.html#91].

Compiling maturin for musl (as done when building the pip wheels) and then checking the platform returns musl, not gnu.

cargo build --target x86_64-unknown-linux-musl
...
poetry run maturin develop
Created target: Linux X86_64, Some(Musl): "x86_64-unknown-linux-musl"

Looks like the fix mentioned will work then