Installation on M1 Mac with Poetry (also happens with pip install)

[Hammad-NobleAI]

Specifications

• OS: Mac OS 12.5, M1 Pro
• SCS Version: 3.2.0 (also tried 3.1.0)
• Compiler: Installed through Pypi

Description

I'm trying to add scs (it's a dependency of another library for our project) to my Poetry environment, but it seems to fail on importing 'distutils.msvccompiler'. However, when I run Python CLI inside the Poetry virtual env and manually import it, there seems to be no issue with importing it. I installed LAPACK and OPENBLAS through brew using the following set of commands:

brew install lapack openblas
export LDFLAGS="-L/opt/homebrew/opt/openblas/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openblas/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig"
export LDFLAGS="-L/opt/homebrew/opt/lapack/lib"
export CPPFLAGS="-I/opt/homebrew/opt/lapack/include"
export CFLAGS=-Wno-error=implicit-function-declaration
export PKG_CONFIG_PATH="/opt/homebrew/opt/lapack/lib/pkgconfig"
export LAPACK=/opt/homebrew/opt/lapack/lib/liblapack.dylib
export BLAS=/opt/homebrew/opt/openblas/lib/libopenblasp-r0.3.20.dylib

How to reproduce

pip install scs or poetry add scs

Output

bash-5.1$ poetry add scs
Using version ^3.2.0 for scs

Updating dependencies
Resolving dependencies... (2.4s)

Writing lock file

Package operations: 1 install, 0 updates, 2 removals

  • Removing osqp (0.6.2.post5)
  • Removing qdldl (0.1.5.post2)
  • Installing scs (3.2.0): Failed

  EnvCommandError

  Command ['/Users/hammadrehman/Library/Caches/pypoetry/virtualenvs/tf-test-J_unPwNU-py3.10/bin/pip', 'install', '--no-deps', 'file:///Users/hammadrehman/Library/Caches/pypoetry/artifacts/c1/90/a8/d43d0f898e899669586bb9fea2b11b0d15280fb60a92fb95473355800f/scs-3.2.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /Users/hammadrehman/Library/Caches/pypoetry/artifacts/c1/90/a8/d43d0f898e899669586bb9fea2b11b0d15280fb60a92fb95473355800f/scs-3.2.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error

    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [15 lines of output]
        Traceback (most recent call last):
          File "/Users/hammadrehman/Library/Caches/pypoetry/virtualenvs/tf-test-J_unPwNU-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
            main()
          File "/Users/hammadrehman/Library/Caches/pypoetry/virtualenvs/tf-test-J_unPwNU-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/Users/hammadrehman/Library/Caches/pypoetry/virtualenvs/tf-test-J_unPwNU-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/private/var/folders/1r/hdg3z4p5315dq3p2nyh2dpk40000gn/T/pip-build-env-p7zqrucv/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=['wheel'])
          File "/private/var/folders/1r/hdg3z4p5315dq3p2nyh2dpk40000gn/T/pip-build-env-p7zqrucv/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
            self.run_setup()
          File "/private/var/folders/1r/hdg3z4p5315dq3p2nyh2dpk40000gn/T/pip-build-env-p7zqrucv/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 2, in <module>
        ModuleNotFoundError: No module named 'distutils.msvccompiler'
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.

  [notice] A new release of pip available: 22.1.2 -> 22.2.2
  [notice] To update, run: pip install --upgrade pip

  at /opt/homebrew/Cellar/poetry/1.1.14/libexec/lib/python3.10/site-packages/poetry/utils/env.py:1195 in _run
      1191│                 output = subprocess.check_output(
      1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1193│                 )
      1194│         except CalledProcessError as e:
    → 1195│             raise EnvCommandError(e, input=input_)
      1196│ 
      1197│         return decode(output)
      1198│ 
      1199│     def execute(self, bin, *args, **kwargs):

Failed to add packages, reverting the pyproject.toml file to its original content.

[bodono]

I'm not sure what's going on here because distutils.msvccompiler is the microsoft visual C++ compiler that you're trying to use. It seems to me like something is broken with your python install. Perhaps it is the Compiler: Installed through Pypi step you mentioned, which is not necessary. I have an M1 MacBook pro and can install SCS using pip on a clean conda env without issue.

[hansrah]

The arm64 wheel is no longer in pypi. I can pip install as I have the wheel in my pip cache but anybody who does not have it in their cache already cannot install it and gets the same error as the OP.

[bodono]

Hm, ok. Was the arm64 mac-os wheel was ever on pypi? The wheels are built and uploaded by this github action and I'm not sure how that could be updated to also upload the arm64 wheels. If anyone knows I would be happy to accept a PR.

On my M1 Macbook pro it appears to successfully use the x86_64 wheel, I'm not sure how, perhaps emulation?

2022-08-16 16:35:52 (base) 0 bodonoghue@bodonoghue-macbookpro2:~:
=> conda activate clean
2022-08-16 16:36:00 (clean) 0 bodonoghue@bodonoghue-macbookpro2:~:
=> pip install scs
Collecting scs
  Using cached scs-3.2.0-cp310-cp310-macosx_10_9_x86_64.whl (11.4 MB)
Collecting scipy>=0.13.2
  Downloading scipy-1.9.0-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl (58.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.4/58.4 MB 42.1 MB/s eta 0:00:00
Collecting numpy>=1.7
  Downloading numpy-1.23.2-cp310-cp310-macosx_10_9_x86_64.whl (18.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.1/18.1 MB 57.3 MB/s eta 0:00:00
Installing collected packages: numpy, scipy, scs
Successfully installed numpy-1.23.2 scipy-1.9.0 scs-3.2.0
2022-08-16 16:36:15 (clean) 0 bodonoghue@bodonoghue-macbookpro2:~:
=> python
Python 3.10.4 (main, Mar 31 2022, 03:38:35) [Clang 12.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
im>>> import scs
scs.__version__
>>> scs.__version__
'3.2.0'

[hansrah]

I have a scs-3.2.0-cp38-cp38-macosx_12_0_arm64.whl file in my pip cache so I assume it was as it worked fine last week. pip attempts to use scs-3.2.0.tar.gz which fails with the same error as the OP's error.

And if I use the same python version and download the cp310 x86 wheel manually I get the following. Not sure if Conda is doing something different to pip.

pip install scs-3.2.0-cp310-cp310-macosx_10_9_x86_64.whl
ERROR: scs-3.2.0-cp310-cp310-macosx_10_9_x86_64.whl is not a supported wheel on this platform.

[Hammad-NobleAI]

It is possible that it might work under emulation, but for our project, emulation is off the table. Just to add to this... if I clone the scs-python repository and build it using python setup.py install it installs with no issues.

[bodono]

Ok looks like building from source is the workaround for M1 macs for now. I'm not sure a better option is available until we can automate building and uploading MacOS arm64 wheels from github actions.