kevinbdsouza
commented
1 month ago Seems like version mismatch between pydantic-core, tokenizers, orjson, nh3 was creating some issues. Resolved by running:
module load rust on compute canada, and
pip install --no-index maturin
pip install pydantic-core==2.18.1 tokenizers==0.15.0 --editable .[dev]
I'm on linux x86_64, with python 3.11.
pip install --editable .[dev]gives maturin failed error, see error log below. I tried changing rust versions (1.77 and 1.79), but get the same maturin failed error. I tried just installing nh3 bypip install nh3, get a similar error. Not sure what exactly is causing this, and any help is appreciated.I have to build from source because
pip install gdm-concordiahas a pandas<=2.0.3 requirement which my cluster doesn't have supported wheels for.Error log
Building wheel for pydantic-core (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for pydantic-core (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [135 lines of output] Running `maturin pep517 build-wheel -i concordia/bin/python3.11 --compatibility off` 📦 Including license file "/tmp/pip-install-l1bftpbg/pydantic-core_3e0bbf1a5ee741ab9a36b9cbd389e1d8/LICENSE" 🍹 Building a mixed python/rust project 🔗 Found pyo3 bindings 🐍 Found CPython 3.11 at concordia/bin/python3.11 📡 Using build options features, bindings from pyproject.toml Compiling target-lexicon v0.12.14 Compiling python3-dll-a v0.2.10 Compiling once_cell v1.19.0 Compiling proc-macro2 v1.0.86 Compiling unicode-ident v1.0.12 Compiling autocfg v1.3.0 Compiling heck v0.5.0 Compiling libc v0.2.155 Compiling version_check v0.9.4 Compiling rustversion v1.0.17 Compiling tinyvec_macros v0.1.1 Compiling static_assertions v1.1.0 Compiling cfg-if v1.0.0 Compiling radium v0.7.0 Compiling tap v1.0.1 Compiling serde v1.0.203 Compiling memchr v2.7.4 Compiling funty v2.0.0 Compiling zerocopy v0.7.34 Compiling unicode-bidi v0.3.15 Compiling equivalent v1.0.1 Compiling indoc v2.0.5 Compiling regex-syntax v0.8.4 Compiling hashbrown v0.14.5 Compiling unindent v0.2.3 Compiling serde_json v1.0.118 Compiling percent-encoding v2.3.1 Compiling smallvec v1.13.2 Compiling itoa v1.0.11 Compiling ryu v1.0.18 Compiling base64 v0.21.7 Compiling tinyvec v1.6.1 Compiling uuid v1.9.1 Compiling lexical-util v0.8.5 Compiling wyz v0.5.1 Compiling num-traits v0.2.19 Compiling memoffset v0.9.1 Compiling ahash v0.8.11 Compiling form_urlencoded v1.2.1 Compiling aho-corasick v1.1.3 Compiling indexmap v2.2.6 Compiling bitvec v1.0.1 Compiling pyo3-build-config v0.22.0 Compiling unicode-normalization v0.1.23 Compiling lexical-parse-integer v0.8.6 Compiling quote v1.0.36 Compiling getrandom v0.2.15 Compiling syn v2.0.68 Compiling lexical-parse-float v0.8.5 Compiling idna v0.5.0 Compiling num-integer v0.1.46 Compiling num-bigint v0.4.6 Compiling url v2.5.2 Compiling regex-automata v0.4.7 Compiling pyo3-ffi v0.22.0 Compiling pyo3-macros-backend v0.22.0 Compiling pyo3 v0.22.0 Compiling jiter v0.5.0 Compiling pydantic-core v2.20.1 (/tmp/pip-install-l1bftpbg/pydantic-core_3e0bbf1a5ee741ab9a36b9cbd389e1d8) Compiling regex v1.10.5 Compiling strum_macros v0.26.4 Compiling serde_derive v1.0.203 Compiling enum_dispatch v0.3.13 error[E0428]: the name `filter_by_rust_version` is defined multiple times --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum_macros-0.26.4/src/macros/from_repr.rs:106:9 | 101 | fn filter_by_rust_version(_: TokenStream) -> TokenStream { | -------------------------------------------------------- previous definition of the value `filter_by_rust_version` here ... 106 | fn filter_by_rust_version(s: TokenStream) -> TokenStream { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `filter_by_rust_version` redefined here | = note: `filter_by_rust_version` must be defined only once in the value namespace of this block error[E0428]: the name `try_from_str` is defined multiple times --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum_macros-0.26.4/src/macros/strings/from_string.rs:182:1 | 171 | / fn try_from_str( 172 | | _name: &proc_macro2::Ident, 173 | | _impl_generics: &syn::ImplGenerics, 174 | | _ty_generics: &syn::TypeGenerics, ... | 178 | | Default::default() 179 | | } | |_- previous definition of the value `try_from_str` here ... 182 | / fn try_from_str( 183 | | name: &proc_macro2::Ident, 184 | | impl_generics: &syn::ImplGenerics, 185 | | ty_generics: &syn::TypeGenerics, ... | 197 | | } 198 | | } | |_^ `try_from_str` redefined here | = note: `try_from_str` must be defined only once in the value namespace of this module error[E0463]: can't find crate for `rustversion` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum_macros-0.26.4/src/macros/from_repr.rs:100:11 | 100 | #[rustversion::before(1.46)] | ^^^^^^^^^^^ can't find crate error[E0463]: can't find crate for `rustversion` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum_macros-0.26.4/src/macros/from_repr.rs:105:11 | 105 | #[rustversion::since(1.46)] | ^^^^^^^^^^^ can't find crate error[E0463]: can't find crate for `rustversion` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum_macros-0.26.4/src/macros/strings/from_string.rs:170:3 | 170 | #[rustversion::before(1.34)] | ^^^^^^^^^^^ can't find crate error[E0463]: can't find crate for `rustversion` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum_macros-0.26.4/src/macros/strings/from_string.rs:181:3 | 181 | #[rustversion::since(1.34)] | ^^^^^^^^^^^ can't find crate Some errors have detailed explanations: E0428, E0463. For more information about an error, try `rustc --explain E0428`. error: could not compile `strum_macros` (lib) due to 6 previous errors warning: build failed, waiting for other jobs to finish... 💥 maturin failed Caused by: Failed to build a native library through cargo Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="concordia/bin/python3.11" PYTHON_SYS_EXECUTABLE="concordia/bin/python3.11" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-l1bftpbg/pydantic-core_3e0bbf1a5ee741ab9a36b9cbd389e1d8/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib"` Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'concordia/bin/python3.11', '--compatibility', 'off'] returned non-zero exit status 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pydantic-core Building wheel for nh3 (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for nh3 (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [183 lines of output] Running `maturin pep517 build-wheel -i concordia/bin/python3.11 --compatibility off` 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7 🐍 Not using a specific python interpreter 📡 Using build options features from pyproject.toml Compiling target-lexicon v0.12.14 Compiling libc v0.2.155 Compiling proc-macro2 v1.0.86 Compiling unicode-ident v1.0.12 Compiling cfg-if v1.0.0 Compiling autocfg v1.3.0 Compiling ppv-lite86 v0.2.17 Compiling once_cell v1.19.0 Compiling siphasher v0.3.11 Compiling parking_lot_core v0.9.10 Compiling new_debug_unreachable v1.0.6 Compiling scopeguard v1.2.0 Compiling smallvec v1.13.2 Compiling serde v1.0.204 Compiling mac v0.1.1 Compiling tinyvec_macros v0.1.1 Compiling precomputed-hash v0.1.1 Compiling utf-8 v0.7.6 Compiling percent-encoding v2.3.1 Compiling unicode-bidi v0.3.15 Compiling heck v0.5.0 Compiling log v0.4.22 Compiling maplit v1.0.2 Compiling indoc v2.0.5 Compiling unindent v0.2.3 Compiling tinyvec v1.7.0 Compiling futf v0.1.5 Compiling form_urlencoded v1.2.1 Compiling phf_shared v0.11.2 Compiling phf_shared v0.10.0 Compiling tendril v0.4.3 Compiling lock_api v0.4.12 Compiling memoffset v0.9.1 Compiling phf v0.11.2 Compiling pyo3-build-config v0.22.0 Compiling quote v1.0.36 Compiling getrandom v0.2.15 Compiling syn v2.0.69 Compiling rand_core v0.6.4 Compiling unicode-normalization v0.1.23 Compiling rand_chacha v0.3.1 Compiling parking_lot v0.12.3 Compiling rand v0.8.5 Compiling idna v0.5.0 Compiling url v2.5.2 Compiling phf_generator v0.10.0 Compiling phf_generator v0.11.2 Compiling pyo3-macros-backend v0.22.0 Compiling pyo3-ffi v0.22.0 Compiling pyo3 v0.22.0 Compiling string_cache_codegen v0.5.2 Compiling phf_codegen v0.11.2 Compiling markup5ever v0.12.1 Compiling string_cache v0.8.7 Compiling html5ever v0.27.0 Compiling ammonia v4.0.0 Compiling pyo3-macros v0.22.0 error[E0463]: can't find crate for `pyo3_macros` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/prelude.rs:28:9 | 28 | pub use pyo3_macros::{pyclass, pyfunction, pymethods, pymodule, FromPyObject}; | ^^^^^^^^^^^ can't find crate error[E0463]: can't find crate for `pyo3_macros` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/lib.rs:478:9 | 478 | pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; | ^^^^^^^^^^^ can't find crate error[E0463]: can't find crate for `pyo3_macros` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/lib.rs:489:9 | 489 | pub use pyo3_macros::pyclass; | ^^^^^^^^^^^ can't find crate error[E0463]: can't find crate for `indoc` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/lib.rs:424:5 | 424 | indoc, // Re-exported for py_run | ^^^^^ can't find crate error[E0432]: unresolved imports `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject`, `crate::FromPyObject` --> /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/array.rs:5:30 | 5 | err::DowncastError, ffi, FromPyObject, IntoPy, Py, PyAny, PyObject, PyResult, Python, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/cell.rs:4:38 | 4 | types::any::PyAnyMethods, Bound, FromPyObject, IntoPy, PyAny, PyObject, PyResult, Python, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/ipaddr.rs:9:21 | 9 | use crate::{intern, FromPyObject, IntoPy, Py, PyAny, PyObject, PyResult, Python, ToPyObject}; | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/map.rs:9:5 | 9 | FromPyObject, IntoPy, PyAny, PyErr, PyObject, Python, ToPyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/num.rs:6:29 | 6 | exceptions, ffi, Bound, FromPyObject, IntoPy, PyAny, PyErr, PyObject, PyResult, Python, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/option.rs:2:56 | 2 | ffi, types::any::PyAnyMethods, AsPyPointer, Bound, FromPyObject, IntoPy, PyAny, PyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/osstr.rs:4:18 | 4 | use crate::{ffi, FromPyObject, IntoPy, PyAny, PyObject, PyResult, Python, ToPyObject}; | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/path.rs:4:18 | 4 | use crate::{ffi, FromPyObject, IntoPy, PyAny, PyObject, PyResult, Python, ToPyObject}; | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/set.rs:11:5 | 11 | FromPyObject, IntoPy, PyAny, PyErr, PyObject, PyResult, Python, ToPyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/string.rs:8:5 | 8 | FromPyObject, IntoPy, Py, PyAny, PyObject, PyResult, Python, ToPyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/conversions/std/time.rs:11:20 | 11 | intern, Bound, FromPyObject, IntoPy, PyAny, PyErr, PyObject, PyResult, Python, ToPyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/impl_/frompyobject.rs:3:38 | 3 | use crate::{exceptions::PyTypeError, FromPyObject, PyAny, PyErr, PyResult, Python}; | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/instance.rs:11:38 | 11 | ffi, AsPyPointer, DowncastError, FromPyObject, IntoPy, PyAny, PyClass, PyClassInitializer, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/pybacked.rs:10:27 | 10 | Bound, DowncastError, FromPyObject, IntoPy, Py, PyAny, PyErr, PyResult, Python, ToPyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/types/boolobject.rs:7:49 | 7 | types::typeobject::PyTypeMethods, Borrowed, FromPyObject, IntoPy, PyAny, PyObject, PyResult, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/types/float.rs:7:51 | 7 | ffi, ffi_ptr_ext::FfiPtrExt, instance::Bound, FromPyObject, IntoPy, PyAny, PyErr, PyObject, | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/types/sequence.rs:14:18 | 14 | use crate::{ffi, FromPyObject, Py, PyTypeCheck, Python, ToPyObject}; | ^^^^^^^^^^^^ | ::: /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.0/src/types/tuple.rs:13:24 | 13 | exceptions, Bound, FromPyObject, IntoPy, Py, PyAny, PyErr, PyObject, PyResult, Python, | ^^^^^^^^^^^^ Some errors have detailed explanations: E0432, E0463. For more information about an error, try `rustc --explain E0432`. error: could not compile `pyo3` (lib) due to 5 previous errors 💥 maturin failed Caused by: Failed to build a native library through cargo Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="concordia/bin/python3.11" PYTHON_SYS_EXECUTABLE="concordia/bin/python3.11" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-l1bftpbg/nh3_ea8f0fc81cb2423490b29f9750768334/Cargo.toml" "--release" "--lib"` Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'concordia/bin/python3.11', '--compatibility', 'off'] returned non-zero exit status 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for nh3 Successfully built gdm-concordia Failed to build pydantic-core nh3 ERROR: Could not build wheels for pydantic-core, nh3, which is required to install pyproject.toml-based projects