RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.25k stars 1.25k forks source link

MacOS wheel build fails when installing NumPy #21830

Closed xuchenhan-tri closed 4 days ago

xuchenhan-tri commented 3 weeks ago

What happened?

mac-arm-ventura-unprovisioned-clang-wheel-nightly-release failed when building Numpy 2.1.0. See output log below.

21829 proposed a temporary fix to upper bound the mac binary Numpy version to 2.0.1.

Version

No response

What operating system are you using?

macOS 13 (Ventura)

What installation option are you using?

No response

Relevant log output

[2024-08-19T20:40:48.053Z]   Preparing metadata (pyproject.toml): started
[2024-08-19T20:41:00.963Z]   Preparing metadata (pyproject.toml): finished with status 'error'
[2024-08-19T20:41:00.963Z]   error: subprocess-exited-with-error
[2024-08-19T20:41:00.963Z]   
[2024-08-19T20:41:00.963Z]   × Preparing metadata (pyproject.toml) did not run successfully.
[2024-08-19T20:41:00.963Z]   │ exit code: 1
[2024-08-19T20:41:00.963Z]   ╰─> [292 lines of output]
[2024-08-19T20:41:00.963Z]       + /opt/drake-wheel-build/python3.11/python/bin/python3.11 /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/vendored-meson/meson/meson.py setup /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/.mesonpy-75i1sxqh -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/.mesonpy-75i1sxqh/meson-python-native-file.ini
[2024-08-19T20:41:00.963Z]       The Meson build system
[2024-08-19T20:41:00.963Z]       Version: 1.4.99
[2024-08-19T20:41:00.963Z]       Source dir: /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe
[2024-08-19T20:41:00.963Z]       Build dir: /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/.mesonpy-75i1sxqh
[2024-08-19T20:41:00.963Z]       Build type: native build
[2024-08-19T20:41:00.963Z]       Project name: NumPy
[2024-08-19T20:41:00.963Z]       Project version: 2.1.0
[2024-08-19T20:41:00.963Z]       C compiler for the host machine: cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
[2024-08-19T20:41:00.963Z]       C linker for the host machine: cc ld64 857.1
[2024-08-19T20:41:00.963Z]       C++ compiler for the host machine: c++ (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
[2024-08-19T20:41:00.963Z]       C++ linker for the host machine: c++ ld64 857.1
[2024-08-19T20:41:00.963Z]       Cython compiler for the host machine: cython (cython 3.0.11)
[2024-08-19T20:41:00.963Z]       Host machine cpu family: aarch64
[2024-08-19T20:41:00.963Z]       Host machine cpu: aarch64
[2024-08-19T20:41:00.963Z]       Program python found: YES (/opt/drake-wheel-build/python3.11/python/bin/python3.11)
[2024-08-19T20:41:00.963Z]       Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 0.29.2
[2024-08-19T20:41:00.963Z]       Run-time dependency python found: YES 3.11
[2024-08-19T20:41:00.963Z]       Has header "Python.h" with dependency python-3.11: YES
[2024-08-19T20:41:00.963Z]       Compiler for C supports arguments -fno-strict-aliasing: YES
[2024-08-19T20:41:00.963Z]       Compiler for C supports arguments -ftrapping-math: YES
[2024-08-19T20:41:00.963Z]       Compiler for C supports link arguments -Wl,-ld_classic: NO
[2024-08-19T20:41:00.963Z]       Message: During parsing cpu-dispatch: The following CPU features were ignored due to platform incompatibility or lack of support:
[2024-08-19T20:41:00.963Z]       "XOP FMA4"
[2024-08-19T20:41:00.963Z]       Test features "NEON NEON_FP16 NEON_VFPV4 ASIMD" : Supported
[2024-08-19T20:41:00.963Z]       Test features "ASIMDHP" : Supported
[2024-08-19T20:41:00.963Z]       Test features "ASIMDFHM" : Supported
[2024-08-19T20:41:00.963Z]       Test features "SVE" : Unsupported due to Compiler fails against the test code of "SVE"
[2024-08-19T20:41:00.963Z]       Configuring npy_cpu_dispatch_config.h using configuration
[2024-08-19T20:41:00.963Z]       Message:
[2024-08-19T20:41:00.963Z]       CPU Optimization Options
[2024-08-19T20:41:00.963Z]         baseline:
[2024-08-19T20:41:00.963Z]           Requested : min
[2024-08-19T20:41:00.963Z]           Enabled   : NEON NEON_FP16 NEON_VFPV4 ASIMD
[2024-08-19T20:41:00.963Z]         dispatch:
[2024-08-19T20:41:00.963Z]           Requested : max -xop -fma4
[2024-08-19T20:41:00.963Z]           Enabled   : ASIMDHP ASIMDFHM
[2024-08-19T20:41:00.963Z]       
[2024-08-19T20:41:00.963Z]       Library m found: YES
[2024-08-19T20:41:00.963Z]       Run-time dependency scipy-openblas found: NO (tried pkgconfig)
[2024-08-19T20:41:00.963Z]       Run-time dependency accelerate found: NO (tried system)
[2024-08-19T20:41:00.963Z]       Found CMake: /opt/homebrew/bin/cmake (3.30.2)
[2024-08-19T20:41:00.964Z]       WARNING: CMake Toolchain: Failed to determine CMake compilers state
[2024-08-19T20:41:00.964Z]       Run-time dependency openblas found: NO (tried pkgconfig, pkgconfig, pkgconfig, system and cmake)
[2024-08-19T20:41:00.964Z]       Run-time dependency flexiblas found: NO (tried pkgconfig, framework and cmake)
[2024-08-19T20:41:00.964Z]       Run-time dependency blis found: NO (tried pkgconfig, framework and cmake)
[2024-08-19T20:41:00.964Z]       Run-time dependency blas found: YES unknown
[2024-08-19T20:41:00.964Z]       ../numpy/meson.build:135: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'blas' custom lookup.
[2024-08-19T20:41:00.964Z]       Message: BLAS symbol suffix:
[2024-08-19T20:41:00.964Z]       Run-time dependency accelerate found: NO (tried system)
[2024-08-19T20:41:00.964Z]       Run-time dependency openblas found: NO (tried pkgconfig, pkgconfig, pkgconfig, system and cmake)
[2024-08-19T20:41:00.964Z]       Run-time dependency flexiblas found: NO (tried pkgconfig, framework and cmake)
[2024-08-19T20:41:00.964Z]       Run-time dependency lapack found: YES unknown
[2024-08-19T20:41:00.964Z]       ../numpy/meson.build:198: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'lapack' custom lookup.
[2024-08-19T20:41:00.964Z]       Checking if "Check atomic builtins without -latomic" : links: NO
[2024-08-19T20:41:00.964Z]       Library atomic found: NO
[2024-08-19T20:41:00.964Z]       Program _build_utils/process_src_template.py found: YES (/opt/drake-wheel-build/python3.11/python/bin/python3.11 /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/numpy/_build_utils/process_src_template.py)
[2024-08-19T20:41:00.964Z]       Program _build_utils/tempita.py found: YES (/opt/drake-wheel-build/python3.11/python/bin/python3.11 /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/numpy/_build_utils/tempita.py)

...

[2024-08-19T20:41:00.966Z]       Build targets in project: 63
[2024-08-19T20:41:00.966Z]       WARNING: Project specifies a minimum meson_version '>=1.2.99' but uses features which were added in newer versions:
[2024-08-19T20:41:00.966Z]        * 1.3.0: {'dep 'blas' custom lookup', 'dep 'lapack' custom lookup'}
[2024-08-19T20:41:00.966Z]       
[2024-08-19T20:41:00.966Z]       NumPy 2.1.0
[2024-08-19T20:41:00.966Z]       
[2024-08-19T20:41:00.966Z]         User defined options
[2024-08-19T20:41:00.966Z]           Native files: /private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/.mesonpy-75i1sxqh/meson-python-native-file.ini
[2024-08-19T20:41:00.966Z]           buildtype   : release
[2024-08-19T20:41:00.966Z]           b_ndebug    : if-release
[2024-08-19T20:41:00.966Z]           b_vscrt     : md
[2024-08-19T20:41:00.966Z]       
[2024-08-19T20:41:00.966Z]       Found ninja-1.12.1 at /opt/homebrew/bin/ninja
[2024-08-19T20:41:00.966Z]       + /opt/homebrew/bin/ninja
[2024-08-19T20:41:00.966Z]       [1/320] Generating 'numpy/_core/libnpymath.a.p/ieee754.c'
[2024-08-19T20:41:00.966Z]       [2/320] Generating 'numpy/_core/libargfunc.dispatch.h_baseline.a.p/npy_sort.h'
[2024-08-19T20:41:00.966Z]       [3/320] Generating 'numpy/_core/libargfunc.dispatch.h_baseline.a.p/arraytypes.h'
[2024-08-19T20:41:00.966Z]       [4/320] Generating numpy/_core/__ufunc_api with a custom command
[2024-08-19T20:41:00.966Z]       FAILED: numpy/_core/__ufunc_api.c numpy/_core/__ufunc_api.h
[2024-08-19T20:41:00.966Z]       /opt/drake-wheel-build/python3.11/python/bin/python3.11 ../numpy/_core/code_generators/generate_ufunc_api.py -o numpy/_core
[2024-08-19T20:41:00.966Z]       Traceback (most recent call last):
[2024-08-19T20:41:00.966Z]         File "/private/var/folders/0h/x72zp8qj45xfftcw9yh2hg_m0000gn/T/pip-install-zmukoya7/numpy_0fcae2117b8943b79311589194dfcdfe/.mesonpy-75i1sxqh/../numpy/_core/code_generators/generate_ufunc_api.py", line 4, in <module>
[2024-08-19T20:41:00.966Z]           import genapi
[2024-08-19T20:41:00.966Z]       ModuleNotFoundError: No module named 'genapi'
[2024-08-19T20:41:00.966Z]       [5/320] Generating numpy/_core/npy_math_internal.h with a custom command
[2024-08-19T20:41:00.966Z]       [6/320] Generating 'numpy/_core/libnpymath.a.p/npy_math_complex.c'
[2024-08-19T20:41:00.966Z]       [7/320] Copying file numpy/__init__.py
[2024-08-19T20:41:00.966Z]       [8/320] Copying file numpy/__init__.pxd
[2024-08-19T20:41:00.966Z]       [9/320] Copying file numpy/__init__.cython-30.pxd
[2024-08-19T20:41:00.966Z]       [10/320] Generating 'numpy/_core/libhighway_qsort.dispatch.h_ASIMD.a.p/npy_sort.h'
[2024-08-19T20:41:00.966Z]       [11/320] Generating 'numpy/_core/libhighway_qsort.dispatch.h_ASIMD.a.p/arraytypes.h'
[2024-08-19T20:41:00.966Z]       [12/320] Generating 'numpy/_core/libargfunc.dispatch.h_baseline.a.p/argfunc.dispatch.c'
[2024-08-19T20:41:00.966Z]       [13/320] Generating numpy/_core/__umath_generated with a custom command
[2024-08-19T20:41:00.966Z]       ninja: build stopped: subcommand failed.
[2024-08-19T20:41:00.966Z]       [end of output]
[2024-08-19T20:41:00.966Z]   
[2024-08-19T20:41:00.966Z]   note: This error originates from a subprocess, and is likely not a problem with pip.
[2024-08-19T20:41:00.966Z] error: metadata-generation-failed
[2024-08-19T20:41:00.966Z] 
[2024-08-19T20:41:00.966Z] × Encountered error while generating package metadata.
[2024-08-19T20:41:00.966Z] ╰─> See above for output.
[2024-08-19T20:41:00.966Z] 
[2024-08-19T20:41:00.966Z] note: This is an issue with the package mentioned above, not pip.
[2024-08-19T20:41:00.966Z] hint: See above for details.
jwnimmer-tri commented 3 weeks ago

Building numpy from source means all of our men are already dead. (In other words, the exact nature of the compilation error is not relevant.)

The important question here is why the build isn't / wasn't using a downloaded wheel in the first place.

Maybe there was just a lag in publishing newer wheels upstream?

SeanCurtis-TRI commented 2 weeks ago

After #21829, we're getting a new problem. Trying to get pip to downgrade a numpy version that was apparently installed by brew. (See, e.g., this CI failure.)

[5:21](https://drakedevelopers.slack.com/archives/C270MN28G/p1724674907776519?thread_ts=1724674598.735349&cid=C270MN28G)
Untitled

[11:29:48 PM]    Attempting uninstall: numpy
[11:29:48 PM]      Found existing installation: numpy 2.1.0
[11:29:48 PM]  error: uninstall-no-record-file
[11:29:48 PM]  
[11:29:48 PM]  × Cannot uninstall numpy 2.1.0
[11:29:48 PM]  ╰─> The package's contents are unknown: no RECORD file was found for numpy.
[11:29:48 PM]  
[11:29:48 PM]  hint: The package was installed by brew. You should check if it can uninstall the package.
[11:29:49 PM]  CMake Error at /Users/admin/workspace/mac-arm-sonoma-unprovisioned-clang-bazel-nightly-release/ci/driver/functions.cmake:100 (message):
[11:29:49 PM]    *** CTest Result: FAILURE BECAUSE PROVISIONING SCRIPT DID NOT COMPLETE
[11:29:49 PM]    SUCCESSFULLY
[11:29:49 PM]  Call Stack (most recent call first):
[11:29:49 PM]    /Users/admin/workspace/mac-arm-sonoma-unprovisioned-clang-bazel-nightly-release/ci/driver/platform.cmake:117 (fatal)
[11:29:49 PM]    /Users/admin/workspace/mac-arm-sonoma-unprovisioned-clang-bazel-nightly-release/ci/ctest_driver_script.cmake:113 (include)
jwnimmer-tri commented 2 weeks ago

The root cause of the failure is that NumPy 2.1.0 did not ship any wheels that are compatible with Ventura (macOS 13). Upstream issue: https://github.com/numpy/numpy/issues/27283. That means for now, we can't ship Drake wheels for Ventura, either.

The action item for this ticket is to watch the upstream issue.

If they decide to add the wheel, then we revert the wheel-builder comment-out fraction of #21845 to re-enable 3.11 support.

If they declare Ventura + cp311 as no longer supported, then we will update our docs to do the same and clean up the comments.

mwoehlke-kitware commented 1 week ago

numpy/numpy#27303 was merged and backported. I'm not sure if that means it's "fixed" or if we're waiting for the next 2.1.x release.

jwnimmer-tri commented 1 week ago

The 2.1.0 on PyPI still lacks the wheel, so it looks like we're going to be waiting for 2.1.1 to happen before we can revert our work-around.

jwnimmer-tri commented 6 days ago

NumPy 2.1.1 has been released, with the missing wheel, so => https://github.com/RobotLocomotion/drake/pull/21883 to revert.