search

Wytamma / snk

Snakemake workflow management system and CLI generation tool
https://snk.wytamma.com
MIT License
31 stars 2 forks source link

JOSS review: Rust-related errors when installing test environment via hatch #85

Closed huddlej closed 2 hours ago

huddlej commented 4 hours ago

Description

When running the test suite with hatch, I got the following compilation error for the Rust-based dependency tiktoken:

$ hatch run test
  error: subprocess-exited-with-error

  × Building wheel for tiktoken (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      copying tiktoken/registry.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      copying tiktoken/__init__.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      copying tiktoken/core.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      copying tiktoken/model.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      copying tiktoken/load.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      copying tiktoken/_educational.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      creating build/lib.macosx-10.13-x86_64-cpython-312/tiktoken_ext
      copying tiktoken_ext/openai_public.py -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken_ext
      running egg_info
      writing tiktoken.egg-info/PKG-INFO
      writing dependency_links to tiktoken.egg-info/dependency_links.txt
      writing requirements to tiktoken.egg-info/requires.txt
      writing top-level names to tiktoken.egg-info/top_level.txt
      reading manifest file 'tiktoken.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'Makefile'
      adding license file 'LICENSE'
      writing manifest file 'tiktoken.egg-info/SOURCES.txt'
      copying tiktoken/py.typed -> build/lib.macosx-10.13-x86_64-cpython-312/tiktoken
      running build_ext
      running build_rust
      error: can't find Rust compiler

      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

      To update pip, run:

          pip install --upgrade pip

      and then retry package installation.

      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tiktoken
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tiktoken)

I installed Rust and retried with a similar hatch command which produced a Rust compilation error:

$ hatch run cov
  error: subprocess-exited-with-error

  × Building wheel for tiktoken (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      copying tiktoken/registry.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      copying tiktoken/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      copying tiktoken/core.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      copying tiktoken/model.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      copying tiktoken/load.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      copying tiktoken/_educational.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      creating build/lib.macosx-10.9-x86_64-cpython-312/tiktoken_ext
      copying tiktoken_ext/openai_public.py -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken_ext
      running egg_info
      writing tiktoken.egg-info/PKG-INFO
      writing dependency_links to tiktoken.egg-info/dependency_links.txt
      writing requirements to tiktoken.egg-info/requires.txt
      writing top-level names to tiktoken.egg-info/top_level.txt
      reading manifest file 'tiktoken.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'Makefile'
      adding license file 'LICENSE'
      writing manifest file 'tiktoken.egg-info/SOURCES.txt'
      copying tiktoken/py.typed -> build/lib.macosx-10.9-x86_64-cpython-312/tiktoken
      running build_ext
      running build_rust
          Updating crates.io index
           Locking 47 packages to latest compatible versions
            Adding bit-set v0.5.3 (latest: v0.8.0)
            Adding bit-vec v0.6.3 (latest: v0.8.0)
            Adding fancy-regex v0.11.0 (latest: v0.14.0)
            Adding indoc v1.0.9 (latest: v2.0.5)
            Adding pyo3 v0.19.2 (latest: v0.22.5)
            Adding pyo3-build-config v0.19.2 (latest: v0.22.5)
            Adding pyo3-ffi v0.19.2 (latest: v0.22.5)
            Adding pyo3-macros v0.19.2 (latest: v0.22.5)
            Adding pyo3-macros-backend v0.19.2 (latest: v0.22.5)
            Adding rustc-hash v1.1.0 (latest: v2.0.0)
            Adding syn v1.0.109 (latest: v2.0.85)
            Adding unindent v0.1.11 (latest: v0.2.3)
      cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --target x86_64-apple-darwin --release -v --features pyo3/extension-module --crate-type cdylib -- -C link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/_tiktoken.cpython-312-darwin.so
         Compiling target-lexicon v0.12.16
         Compiling autocfg v1.4.0
         Compiling once_cell v1.20.2
         Compiling proc-macro2 v1.0.89
         Compiling unicode-ident v1.0.13
         Compiling libc v0.2.161
         Compiling syn v1.0.109
         Compiling memchr v2.7.4
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name autocfg --edition=2015 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/autocfg-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=b893af90c314e9d6 -C extra-filename=-b893af90c314e9d6 --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name build_script_build --edition=2018 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/target-lexicon-0.12.16/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="default"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("arch_zkasm", "default", "serde", "serde_support", "std"))' -C metadata=05cd1448826cbdf1 -C extra-filename=-05cd1448826cbdf1 --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/build/target-lexicon-05cd1448826cbdf1 -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name once_cell --edition=2021 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/once_cell-1.20.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="race"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("alloc", "atomic-polyfill", "critical-section", "default", "parking_lot", "portable-atomic", "race", "std", "unstable"))' -C metadata=8611d313615a6333 -C extra-filename=-8611d313615a6333 --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name unicode_ident --edition=2018 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/unicode-ident-1.0.13/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=86cf18140559121b -C extra-filename=-86cf18140559121b --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name build_script_build --edition=2015 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libc-0.2.161/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="default"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("align", "const-extern-fn", "default", "extra_traits", "rustc-dep-of-std", "rustc-std-workspace-core", "std", "use_std"))' -C metadata=0ef8d401971a71b6 -C extra-filename=-0ef8d401971a71b6 --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/build/libc-0ef8d401971a71b6 -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name build_script_build --edition=2021 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.89/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="proc-macro"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("default", "nightly", "proc-macro", "span-locations"))' -C metadata=398a93ff578e796a -C extra-filename=-398a93ff578e796a --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/build/proc-macro2-398a93ff578e796a -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name memchr --edition=2021 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/memchr-2.7.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("alloc", "compiler_builtins", "core", "default", "libc", "logging", "rustc-dep-of-std", "std", "use_std"))' -C metadata=29a59d0a6bcf484f -C extra-filename=-29a59d0a6bcf484f --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/x86_64-apple-darwin/release/deps -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
           Running `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name build_script_build --edition=2018 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("clone-impls", "default", "derive", "extra-traits", "fold", "full", "parsing", "printing", "proc-macro", "quote", "test", "visit", "visit-mut"))' -C metadata=0bc609dc0d92f870 -C extra-filename=-0bc609dc0d92f870 --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/build/syn-0bc609dc0d92f870 -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow`
      error[E0463]: can't find crate for `core`
        |
        = note: the `x86_64-apple-darwin` target may not be installed
        = help: consider downloading the target with `rustup target add x86_64-apple-darwin`

      For more information about this error, try `rustc --explain E0463`.
      error: could not compile `memchr` (lib) due to 1 previous error

      Caused by:
        process didn't exit successfully: `/Users/jhuddlesfredhutch.org/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name memchr --edition=2021 /Users/jhuddlesfredhutch.org/.cargo/registry/src/index.crates.io-6f17d22bba15001f/memchr-2.7.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("alloc", "compiler_builtins", "core", "default", "libc", "logging", "rustc-dep-of-std", "std", "use_std"))' -C metadata=29a59d0a6bcf484f -C extra-filename=-29a59d0a6bcf484f --out-dir /private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -C strip=debuginfo -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/x86_64-apple-darwin/release/deps -L dependency=/private/var/folders/gc/yfwm_np95nx55dc62kt3jx240000gp/T/pip-install-2p88c_eu/tiktoken_386946f2ed8c4af0943be18354f0dc2f/target/release/deps --cap-lints allow` (exit status: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --target x86_64-apple-darwin --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/_tiktoken.cpython-312-darwin.so'` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tiktoken
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tiktoken)

It seems like the tiktoken dependency is only a child dependency of the write-the snk dependency, so I tried removing write-the from the project's dependencies list and running hatch run test again. This change allowed me to run the test suite.

Since write-the is specifically needed for building the documentation, what if it was part of a separate optional dependency group in the pyproject file? This would allow contributors to run the test suite without installing Rust or working around the tiktoken dependency.

Is there is another better way to work around this issue?

Motivated by https://github.com/openjournals/joss-reviews/issues/7410

Environment

Wytamma commented 2 hours ago

Thanks @huddlej write-the is not required will remove it from dev-dependency.

Wytamma commented 2 hours ago

Solved in #86