search

pydantic / pydantic-core

Core validation logic for pydantic written in rust
MIT License
1.43k stars 241 forks source link

Fail to build pydantic_core on OS X 10.11 with Python 3.10 #1295

Open BLKSerene opened 5 months ago

BLKSerene commented 5 months ago

Hi, pydantic_core >= 2.16 only provides wheels for macOS >= 10.12 and it would fail when compiling from source on OS X 10.11 with Python 3.10.11:

pip3 install --upgrade pydantic==2.7.1

Requirement already satisfied: pydantic in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (2.5.3)
Collecting pydantic
  Using cached pydantic-2.7.1-py3-none-any.whl.metadata (107 kB)
Requirement already satisfied: annotated-types>=0.4.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pydantic) (0.6.0)
Collecting pydantic-core==2.18.2 (from pydantic)
  Using cached pydantic_core-2.18.2.tar.gz (383 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.6.1 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pydantic) (4.11.0)
Using cached pydantic-2.7.1-py3-none-any.whl (409 kB)
Building wheels for collected packages: pydantic-core
  Building wheel for pydantic-core (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pydantic-core (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Running `maturin pep517 build-wheel -i /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 --compatibility off`
      📦 Including license file "/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-a3iwsm3w/pydantic-core_1e29f10ace204ee99125b93a432e3171/LICENSE"
      🍹 Building a mixed python/rust project
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.10 at /Library/Frameworks/Python.framework/Versions/3.10/bin/python3
      📡 Using build options features, bindings from pyproject.toml
      💻 Using `MACOSX_DEPLOYMENT_TARGET=10.7` for x86_64-apple-darwin by default
      warning: unused manifest key `lints` (may be supported in a future version)

      this Cargo does not support nightly features, but if you
      switch to nightly channel you can pass
      `-Zlints` to enable this feature.
      error: package `pydantic-core v2.18.2 (/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-a3iwsm3w/pydantic-core_1e29f10ace204ee99125b93a432e3171)` cannot be built because it requires rustc 1.76 or newer, while the currently active rustc version is 1.72.1

      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `MACOSX_DEPLOYMENT_TARGET="10.7" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.10-64bit" PYO3_PYTHON="/Library/Frameworks/Python.framework/Versions/3.10/bin/python3" PYTHON_SYS_EXECUTABLE="/Library/Frameworks/Python.framework/Versions/3.10/bin/python3" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-a3iwsm3w/pydantic-core_1e29f10ace204ee99125b93a432e3171/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/pydantic_core._pydantic_core.cpython-310-darwin.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Library/Frameworks/Python.framework/Versions/3.10/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pydantic-core
Failed to build pydantic-core
ERROR: Could not build wheels for pydantic-core, which is required to install pyproject.toml-based projects

pip3 install --upgrade pydantic==2.6.4

Collecting pydantic==2.6.4
  Using cached pydantic-2.6.4-py3-none-any.whl.metadata (85 kB)
Requirement already satisfied: annotated-types>=0.4.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pydantic==2.6.4) (0.6.0)
Collecting pydantic-core==2.16.3 (from pydantic==2.6.4)
  Using cached pydantic_core-2.16.3.tar.gz (368 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.6.1 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from pydantic==2.6.4) (4.11.0)
Using cached pydantic-2.6.4-py3-none-any.whl (394 kB)
Building wheels for collected packages: pydantic-core
  Building wheel for pydantic-core (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pydantic-core (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [155 lines of output]
      Running `maturin pep517 build-wheel -i /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 --compatibility off`
      📦 Including license file "/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-50dvwsdn/pydantic-core_1a7e9b4858b04882a785d743f3156b27/LICENSE"
      🍹 Building a mixed python/rust project
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.10 at /Library/Frameworks/Python.framework/Versions/3.10/bin/python3
      📡 Using build options features, bindings from pyproject.toml
      💻 Using `MACOSX_DEPLOYMENT_TARGET=10.7` for x86_64-apple-darwin by default
      warning: unused manifest key `lints` (may be supported in a future version)

      this Cargo does not support nightly features, but if you
      switch to nightly channel you can pass
      `-Zlints` to enable this feature.
         Compiling autocfg v1.1.0
         Compiling proc-macro2 v1.0.76
         Compiling target-lexicon v0.12.9
         Compiling unicode-ident v1.0.10
         Compiling quote v1.0.35
         Compiling libc v0.2.147
         Compiling python3-dll-a v0.2.9
         Compiling syn v2.0.48
         Compiling pyo3-build-config v0.20.2
         Compiling once_cell v1.18.0
         Compiling version_check v0.9.4
         Compiling cfg-if v1.0.0
         Compiling heck v0.4.1
         Compiling static_assertions v1.1.0
         Compiling lexical-util v0.8.5
         Compiling ahash v0.8.7
         Compiling num-traits v0.2.16
         Compiling getrandom v0.2.10
         Compiling pyo3-ffi v0.20.2
         Compiling num-integer v0.1.45
         Compiling lock_api v0.4.10
         Compiling parking_lot_core v0.9.8
         Compiling zerocopy v0.7.32
         Compiling rustversion v1.0.13
         Compiling num-bigint v0.4.4
         Compiling memoffset v0.9.0
         Compiling tinyvec_macros v0.1.1
         Compiling scopeguard v1.1.0
         Compiling allocator-api2 v0.2.16
         Compiling smallvec v1.11.2
         Compiling hashbrown v0.14.3
         Compiling tinyvec v1.6.0
         Compiling lexical-parse-integer v0.8.6
         Compiling lexical-write-integer v0.8.5
         Compiling pyo3-macros-backend v0.20.2
         Compiling pyo3 v0.20.2
         Compiling serde v1.0.195
         Compiling memchr v2.6.3
         Compiling aho-corasick v1.0.2
         Compiling pyo3-macros v0.20.2
         Compiling lexical-write-float v0.8.5
         Compiling lexical-parse-float v0.8.5
         Compiling strum_macros v0.25.3
         Compiling unicode-normalization v0.1.22
         Compiling parking_lot v0.12.1
         Compiling serde_derive v1.0.195
         Compiling percent-encoding v2.3.1
         Compiling unindent v0.2.3
         Compiling unicode-bidi v0.3.13
         Compiling indoc v2.0.4
         Compiling equivalent v1.0.1
         Compiling regex-syntax v0.8.2
         Compiling serde_json v1.0.109
         Compiling regex-automata v0.4.3
         Compiling indexmap v2.0.0
         Compiling idna v0.5.0
         Compiling form_urlencoded v1.2.1
         Compiling strum v0.25.0
         Compiling lexical-core v0.8.5
         Compiling pydantic-core v2.16.3 (/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-50dvwsdn/pydantic-core_1a7e9b4858b04882a785d743f3156b27)
         Compiling itoa v1.0.8
         Compiling ryu v1.0.14
         Compiling jiter v0.0.6
         Compiling speedate v0.13.0
         Compiling url v2.5.0
         Compiling regex v1.10.2
         Compiling enum_dispatch v0.3.12
         Compiling base64 v0.21.7
         Compiling uuid v1.6.1
      error[E0446]: crate-private type `extra::Extra<'_>` in public interface
         --> src/serializers/fields.rs:149:5
          |
      149 | /     pub fn main_to_python<'py>(
      150 | |         &self,
      151 | |         py: Python<'py>,
      152 | |         main_iter: impl Iterator<Item = PyResult<(&'py PyAny, &'py PyAny)>>,
      ...   |
      155 | |         extra: Extra,
      156 | |     ) -> PyResult<&'py PyDict> {
          | |______________________________^ can't leak crate-private type
          |
         ::: src/serializers/extra.rs:73:1
          |
      73  |   pub(crate) struct Extra<'a> {
          |   --------------------------- `extra::Extra<'_>` declared as crate-private

      error[E0446]: crate-private type `extra::Extra<'_>` in public interface
         --> src/serializers/fields.rs:214:5
          |
      214 | /     pub fn main_serde_serialize<'py, S: serde::ser::Serializer>(
      215 | |         &self,
      216 | |         main_iter: impl Iterator<Item = PyResult<(&'py PyAny, &'py PyAny)>>,
      217 | |         expected_len: usize,
      ...   |
      221 | |         extra: Extra,
      222 | |     ) -> Result<S::SerializeMap, S::Error> {
          | |__________________________________________^ can't leak crate-private type
          |
         ::: src/serializers/extra.rs:73:1
          |
      73  |   pub(crate) struct Extra<'a> {
          |   --------------------------- `extra::Extra<'_>` declared as crate-private

      error[E0446]: crate-private type `extra::Extra<'_>` in public interface
         --> src/serializers/fields.rs:260:5
          |
      260 | /     pub fn add_computed_fields_python(
      261 | |         &self,
      262 | |         model: Option<&PyAny>,
      263 | |         output_dict: &PyDict,
      ...   |
      266 | |         extra: &Extra,
      267 | |     ) -> PyResult<()> {
          | |_____________________^ can't leak crate-private type
          |
         ::: src/serializers/extra.rs:73:1
          |
      73  |   pub(crate) struct Extra<'a> {
          |   --------------------------- `extra::Extra<'_>` declared as crate-private

      error[E0446]: crate-private type `extra::Extra<'_>` in public interface
         --> src/serializers/fields.rs:277:5
          |
      277 | /     pub fn add_computed_fields_json<S: serde::ser::Serializer>(
      278 | |         &self,
      279 | |         model: Option<&PyAny>,
      280 | |         map: &mut S::SerializeMap,
      ...   |
      283 | |         extra: &Extra,
      284 | |     ) -> Result<(), S::Error> {
          | |_____________________________^ can't leak crate-private type
          |
         ::: src/serializers/extra.rs:73:1
          |
      73  |   pub(crate) struct Extra<'a> {
          |   --------------------------- `extra::Extra<'_>` declared as crate-private

      For more information about this error, try `rustc --explain E0446`.
      error: could not compile `pydantic-core` (lib) due to 4 previous errors
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `MACOSX_DEPLOYMENT_TARGET="10.7" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.10-64bit" PYO3_PYTHON="/Library/Frameworks/Python.framework/Versions/3.10/bin/python3" PYTHON_SYS_EXECUTABLE="/Library/Frameworks/Python.framework/Versions/3.10/bin/python3" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-50dvwsdn/pydantic-core_1a7e9b4858b04882a785d743f3156b27/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/pydantic_core._pydantic_core.cpython-310-darwin.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Library/Frameworks/Python.framework/Versions/3.10/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pydantic-core
Failed to build pydantic-core
ERROR: Could not build wheels for pydantic-core, which is required to install pyproject.toml-based projects
davidhewitt commented 5 months ago

The hint here is already in your build output:

(/private/var/folders/_n/8qwtj23j0tx0bdy7ftvpg63m0000gn/T/pip-install-a3iwsm3w/pydantic-core_1e29f10ace204ee99125b93a432e3171)` cannot be built because it requires rustc 1.76 or newer, while the currently active rustc version is 1.72.1

If you update your Rust compiler (probably with rustup update) and retry, it should work.

BLKSerene commented 5 months ago

@davidhewitt Unfortunately, starting from Rust 1.74, the minimum supported macOS platform is macOS 10.12 (see here), so the latest Rust compiler supporting OS X 10.11 is Rust 1.73. I tried installing the latest Rust using the standalone installer, but that did not work.

If pydantic chooses to drop support for macOS < 10.12 (Python 3 still supports far back to macOS 10.9), it would be better to explicitly state that in the doc.

davidhewitt commented 5 months ago

That's a great point. I think we should consider downgrading our minimum Rust to 1.73, but this may not be trivial.

davidhewitt commented 5 months ago

^ reflecting on the above, I think downgrading is the right thing, but I think the reality is that extending support backwards to old macOS versions is not a high priority for me right now. If anyone is interested in helping downgrade, a PR is very welcome.

BLKSerene commented 5 months ago

Could you please give some hints about how to contribute on this specific downgrading issue?

davidhewitt commented 5 months ago

The idea would be that you can check the compile against Rust 1.73 (for example) by running cargo +1.73 check in this directory.

Or you could set the rust-version in Cargo.toml to 1.73 and push a PR, which will run the checks on the pipeline here.

I expect that there are a couple of language features and stdlib APIs which we use which will need replacing. If it's not clear how to replace them, perhaps post back here with examples and I can give suggestions.

tiran commented 5 months ago

Rust 1.73 is missing some features. MSRV can be lowered to 1.75, but not below. See #1315

davidhewitt commented 5 months ago

I think 1.73 would still be reachable but it'd be a lot more work.