Library can't be installed for Python 3.12 on Darwin (macos with Apple Silicon)

jonadaly commented 5 months ago

Specifications

OS: MacOS Sonoma 14.4.1 (23E224)
SCS version: 3.2.4.post1
Python version: 3.12.2
Numpy version: 1.26.4
Scipy version: 1.13.0
Output of get_info print commands as above (if applicable):

Doesn't work on python 3.12, but here's the output from 3.11:

>>> print(get_info("blas_opt"))  # best blas install
{'extra_compile_args': ['-I/System/Library/Frameworks/vecLib.framework/Headers'], 'extra_link_args': ['-Wl,-framework', '-Wl,Accelerate'], 'define_macros': [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None), ('ACCELERATE_NEW_LAPACK', None)]}
>>> print(get_info("lapack_opt"))  # best lapack install
{'extra_compile_args': ['-I/System/Library/Frameworks/vecLib.framework/Headers'], 'extra_link_args': ['-Wl,-framework', '-Wl,Accelerate'], 'define_macros': [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None), ('ACCELERATE_NEW_LAPACK', None)]}
>>> print(get_info("blas"))  # fall back blas install
{}
>>> print(get_info("lapack"))  # fall back lapack install
{}

Data file location dumped with write_data_filename (if applicable): N/A

Description

I can't seem to install scs on an Apple Silicon (M1) macbook due to a missing dependency Accelerate.

How to reproduce

I have openblas installed, and have tried installing while explicitly setting the path to it:

brew install openblas
OPENBLAS="$(brew --prefix openblas)" pip install scs

Additional information

This line of the meson build appears to be the source of the problem.

Does this library support installation on Apple Silicon with the latest version of Python? I can see in the CI that we don't try with the macos-14 runner.

Output

Here's the output from the pip install scs command on a fresh venv:

$ OPENBLAS="$(brew --prefix openblas)" pip install scs
Collecting scs
  Using cached scs-3.2.4.post1.tar.gz (1.5 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      + meson setup /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-5o8e458r/scs_2ee27b829fe4411a878f32b4bd919ee7 /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-5o8e458r/scs_2ee27b829fe4411a878f32b4bd919ee7/.mesonpy-1ud88kl6 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-5o8e458r/scs_2ee27b829fe4411a878f32b4bd919ee7/.mesonpy-1ud88kl6/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.0
      Source dir: /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-5o8e458r/scs_2ee27b829fe4411a878f32b4bd919ee7
      Build dir: /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-5o8e458r/scs_2ee27b829fe4411a878f32b4bd919ee7/.mesonpy-1ud88kl6
      Build type: native build
      Project name: scs
      Project version: undefined
      C compiler for the host machine: cc (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.3.9.4)")
      C linker for the host machine: cc ld64 1053.12
      Host machine cpu family: aarch64
      Host machine cpu: aarch64
      Program python found: YES (/Users/jondaly/.pyenv/versions/3.12.2/envs/scs-python/bin/python3.12)
      Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 0.29.2
      Found CMake: /opt/homebrew/bin/cmake (3.29.3)
      Run-time dependency accelerate found: NO (tried pkgconfig, framework and cmake)

      ../meson.build:8:15: ERROR: Dependency "Accelerate" not found, tried pkgconfig, framework and cmake

      A full log can be found at /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-5o8e458r/scs_2ee27b829fe4411a878f32b4bd919ee7/.mesonpy-1ud88kl6/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

bodono commented 5 months ago

Hi Jon, thanks for posting this. I will take a look. In the meantime I'm tagging @isuruf and @enzbus who might have a quick solution.

enzbus commented 5 months ago

Just a note, that line might have been modified by a PR 2 weeks ago but you're installing a pip package released 6 months ago.

I can't reproduce your issue on mac m1, for me

pip install --no-cache-dir --no-binary :all: scs

in a fresh environment works without touching the OPENBLAS environment variable nor installing openblas.

enzbus commented 5 months ago

Edit:

I suspect you guys are missing this

xcode-select --install

Agreed that pre-built wheels should be available, @bodono you might have to look into CIBuildWheel...

jonadaly commented 5 months ago

Thanks @enzbus for the responses - I have the xcode-select tools installed already.

The no-cache command you suggested produces a different error:

$ pip install --no-cache-dir --no-binary :all: scs
Collecting scs
  Downloading scs-3.2.4.post1.tar.gz (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 5.7 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [120 lines of output]
      Collecting meson-python
        Downloading meson_python-0.16.0.tar.gz (82 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 kB 2.5 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting oldest-supported-numpy
        Downloading oldest-supported-numpy-2023.12.21.tar.gz (5.2 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting meson>=1.2.3 (from meson-python)
        Using cached meson-1.4.0-py3-none-any.whl
      Collecting packaging>=19.0 (from meson-python)
        Using cached packaging-24.0-py3-none-any.whl
      Collecting pyproject-metadata>=0.7.1 (from meson-python)
        Using cached pyproject_metadata-0.8.0-py3-none-any.whl
      Collecting numpy==1.26.2 (from oldest-supported-numpy)
        Downloading numpy-1.26.2.tar.gz (15.7 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 8.3 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: still running...
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        × Preparing metadata (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [66 lines of output]
            + /Users/jondaly/.pyenv/versions/3.12.3/envs/scs-python/bin/python3.12 /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/vendored-meson/meson/meson.py setup /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4 /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/.mesonpy-bg3h8okn/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/.mesonpy-bg3h8okn/build/meson-python-native-file.ini
            The Meson build system
            Version: 1.2.99
            Source dir: /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4
            Build dir: /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/.mesonpy-bg3h8okn/build
            Build type: native build
            Project name: NumPy
            Project version: 1.26.2
            C compiler for the host machine: cc (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.3.9.4)")
            C linker for the host machine: cc ld64 1053.12
            C++ compiler for the host machine: c++ (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.3.9.4)")
            C++ linker for the host machine: c++ ld64 1053.12
            Cython compiler for the host machine: cython (cython 3.0.10)
            Host machine cpu family: aarch64
            Host machine cpu: aarch64
            Program python found: YES (/Users/jondaly/.pyenv/versions/3.12.3/envs/scs-python/bin/python3.12)
            Found pkg-config: /opt/homebrew/bin/pkg-config (0.29.2)
            Run-time dependency python found: YES 3.12
            Has header "Python.h" with dependency python-3.12: YES
            Compiler for C supports arguments -fno-strict-aliasing: NO
            Compiler for C supports arguments -ftrapping-math: NO
            Compiler for C supports link arguments -Wl,-ld_classic: NO
            Message: During parsing cpu-dispatch: The following CPU features were ignored due to platform incompatibility or lack of support:
            "XOP FMA4"
            Test features "NEON NEON_FP16 NEON_VFPV4 ASIMD" : Supported
            Test features "ASIMDHP" : Unsupported due to Arguments "-march=armv8.2-a+fp16" are not supported
            Test features "ASIMDFHM" : Unsupported due to Implied feature "ASIMDHP" is not supported
            Configuring npy_cpu_dispatch_config.h using configuration
            Message:
            CPU Optimization Options
              baseline:
                Requested : min
                Enabled   : NEON NEON_FP16 NEON_VFPV4 ASIMD
              dispatch:
                Requested : max -xop -fma4
                Enabled   :

            Library m found: NO
            Run-time dependency scipy-openblas found: NO (tried pkgconfig)
            Run-time dependency mkl found: NO (tried pkgconfig and system)
            Run-time dependency mkl found: NO (tried pkgconfig and system)
            Run-time dependency accelerate found: YES
            ../../numpy/meson.build:124: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'accelerate' custom lookup.
            Message: BLAS symbol suffix: $NEWLAPACK
            Program _build_utils/process_src_template.py found: YES (/Users/jondaly/.pyenv/versions/3.12.3/envs/scs-python/bin/python3.12 /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/numpy/_build_utils/process_src_template.py)
            Program _build_utils/tempita.py found: YES (/Users/jondaly/.pyenv/versions/3.12.3/envs/scs-python/bin/python3.12 /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/numpy/_build_utils/tempita.py)
            Configuring __config__.py using configuration
            Checking for size of "short" : -1
            Checking for size of "int" : -1
            Checking for size of "long" : -1
            Checking for size of "long long" : -1
            Checking for size of "float" : -1
            Checking for size of "double" : -1
            Checking for size of "long double" : -1
            Checking for size of "off_t" : -1
            Checking for size of "Py_intptr_t" with dependency python-3.12: -1
            Checking for size of "PY_LONG_LONG" with dependency python-3.12: -1
            Has header "complex.h" : YES
            Checking for type "complex float" : NO
            Checking for type "complex double" : NO
            Checking for type "complex long double" : NO
            Checking for function "sin" with dependency -lm: NO

            ../../numpy/core/meson.build:162:4: ERROR: Problem encountered: Function `sin` not found

            A full log can be found at /private/var/folders/zh/c5mj4lqj3_zfd439bg37n7hc0000gn/T/pip-install-huz3b1aw/numpy_5780e62ef5e8446aaacd908cc8128df4/.mesonpy-bg3h8okn/build/meson-logs/meson-log.txt
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

enzbus commented 5 months ago

You probably need a few more header libraries to build also numpy and scipy. Regarding your initial issue, not clear why Accelerate is not visible to pkg-config, if you have xcode installed. Anyways, another fix might be as simple as adding the macos-latest image (looking here https://github.com/actions/runner-images) to the yaml file that runs in CI and then @bodono can trigger a release build. Go on and make a PR?

bodono commented 5 months ago

OK this PR should build and release macos 14 wheels if it looks good to you. It is linking against accelerate.

jonadaly commented 5 months ago

@bodono are you planning to trigger a release / upload new wheels to pypi? 👀

bodono commented 5 months ago

Ok, I finally managed to do that. Can you let me know if it works for you?

jonadaly commented 5 months ago

@bodono it works! Thanks for your efforts 🙏

bodono / scs-python