Not building on Mac OS MacBook Pro with Apple M2 Max chip

[PeterNSteinmetz]

I am trying to build a copy of cffi for the Mac. I downloaded the latest source package. move to that directory and execute `python -m build'. It complains about the deployment target as being 13 as follows:

Installing packages in isolated environment... (wheel)
* Building wheel...
clang-6.0: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=13'
clang-6.0: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=13'

    No working compiler found, or bogus compiler options passed to
    the compiler from Python's standard "distutils" module.  See
    the error messages above.  Likely, the problem is not related
    to CFFI but generic to the setup.py of any Python package that
    tries to compile C code.  (Hints: on OS/X 10.8, for errors about
    -mno-fused-madd see http://stackoverflow.com/questions/22313407/
    Otherwise, see https://wiki.python.org/moin/CompLangPython or
    the IRC channel #python on irc.libera.chat.)

    Trying to continue anyway.  If you are trying to install CFFI from
    a build done in a different context, you can ignore this warning.

running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-13-x86_64-cpython-310
creating build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/_imp_emulation.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/backend_ctypes.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/error.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/setuptools_ext.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/_shimmed_dist_utils.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/__init__.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/cffi_opcode.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/vengine_gen.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/pkgconfig.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/model.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/ffiplatform.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/api.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/vengine_cpy.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/commontypes.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/lock.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/recompiler.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/cparser.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/verifier.py -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/_cffi_include.h -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/parse_c_type.h -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/_embedding.h -> build/lib.macosx-13-x86_64-cpython-310/cffi
copying src/cffi/_cffi_errors.h -> build/lib.macosx-13-x86_64-cpython-310/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.macosx-13-x86_64-cpython-310
creating build/temp.macosx-13-x86_64-cpython-310/src
creating build/temp.macosx-13-x86_64-cpython-310/src/c
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -DFFI_BUILDING=1 -I/usr/local/Cellar/libffi/3.4.4/include -I/private/var/folders/b1/0rcybhys4051l25g6yvknchh00010n/T/build-env-b3adn8di/include -I/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c src/c/_cffi_backend.c -o build/temp.macosx-13-x86_64-cpython-310/src/c/_cffi_backend.o -iwithsysroot/usr/include/ffi
clang-6.0: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=13'
error: command '/usr/local/clang6/bin/clang' failed with exit code 1

The 14.2 SDK is installed and in fact /Library/Developer/CommandLineTools/SDKs/MacOSx.sdk -> MacOSX14.2.sdk.

So it is rather unclear to me what aspect of the build configuration is forcing this -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk flag to the clang compiler and why it is trying to build for build/lib.macosx-13-x86_64-cpython-310 in the first place.

Any suggestions on what needs to be fixed here.

Some background. xcrun --show-sdk-path gives /Library/Developer/CommandLineTools/SDKS/MacOSX14.2.sdk though xcrun --sdk macosx --show-sdk-path gives /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk which I suppose may be the source of the trouble.

[arigo]

@nitzmahone Can you help?

[PeterNSteinmetz]

This is apparently blocking for the rpy2 project to run on the newer apple arm chips and I imagine also for any other projects which use cffi.

[PeterNSteinmetz]

Here is a description of my trying to get an arm64 version installed on MacOS 10.4.2. From the rpy2 issue at https://github.com/rpy2/rpy2/issues/1083

The normal install provides an x86 version:

pip install --prefer-binary cffi
Collecting cffi
  Downloading cffi-1.16.0-cp310-cp310-macosx_10_9_x86_64.whl.metadata (1.5 kB)
Requirement already satisfied: pycparser in ./lib/python3.10/site-packages (from cffi) (2.21)
Downloading cffi-1.16.0-cp310-cp310-macosx_10_9_x86_64.whl (182 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 182.5/182.5 kB 6.3 MB/s eta 0:00:00
Installing collected packages: cffi
Successfully installed cffi-1.16.0

To resolve the question of obtaining an appropriate arm64 binary I have been trying to pip install several versions from https://pypi.org/project/cffi/#files . None of these seem to work.

pip install ~/Downloads/cffi-1.16.0-cp312-cp312-macosx_11_0_arm64.whl 
ERROR: cffi-1.16.0-cp312-cp312-macosx_11_0_arm64.whl is not a supported wheel on this platform.

pip install ~/Downloads/cffi-1.16.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 
ERROR: cffi-1.16.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform.

Thus it appears that one can get an x86 for this os but not an arm64. Which would be consistent with not being able to build it from source on this os and platform.

Is there another one I should try or a different pip command to do a search?

[PeterNSteinmetz]

I noticed that the setup.py when sys.platform=darwin uses homebrew. So I updated homebrew which I had not done since upgrading to MacOS 14.2.1. That all went ok but cffi still does not build and now gives:

cffi-main peter$ python -m build                                                                                                                             * Creating venv isolated environment...                                                                                                                                         * Installing packages in isolated environment... (setuptools >= 66.1)                                                                                                           
* Getting build dependencies for sdist...
clang-6.0: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=14'
clang-6.0: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=14'

    No working compiler found, or bogus compiler options passed to
    the compiler from Python's standard "distutils" module.  See
    the error messages above.  Likely, the problem is not related
    to CFFI but generic to the setup.py of any Python package that
    tries to compile C code.  (Hints: on OS/X 10.8, for errors about
    -mno-fused-madd see http://stackoverflow.com/questions/22313407/
    Otherwise, see https://wiki.python.org/moin/CompLangPython or
    the IRC channel #python on irc.libera.chat.)

    Trying to continue anyway.  If you are trying to install CFFI from
    a build done in a different context, you can ignore this warning.

brew info cffi now gives

brew info cffi
==> cffi: stable 1.16.0 (bottled)
C Foreign Function Interface for Python
https://cffi.readthedocs.io/en/latest/
/usr/local/Cellar/cffi/1.16.0_1 (69 files, 1.1MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-12-29 at 15:22:03
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/c/cffi.rb
License: MIT
==> Dependencies
Build: python@3.11 ✔, python@3.12 ✔
Required: pycparser ✔, python-setuptools ✔
==> Analytics
install: 37,095 (30 days), 162,493 (90 days), 342,275 (365 days)
install-on-request: 16,580 (30 days), 54,069 (90 days), 56,191 (365 days)
build-error: 0 (30 days)

so it appears to be installed.

I notice that the formula seems to have a sonoma arm64 as in the line

    sha256 cellar: :any_skip_relocation, arm64_sonoma:   "2ffe080b9d696ac03d1229276be008dea6f403d395b3c2b3ca9bffa78e50e2f8"

Here is what seems to be installed

ls /usr/local/Cellar/cffi/1.16.0_1/lib/python3.12/site-packages
_cffi_backend.cpython-312-darwin.so cffi                                cffi-1.16.0.dist-info

Same thing happens after building from source with brew install -s cffi. One gets an x86 binary .so library.

[PeterNSteinmetz]

Also noticed that brew doctor said to reinstall brew because I have been using it for a long time and it was installed in /usr/local which only contains the x86 binaries. So I reinstalled and all packages and then a brew install cffi installs a arm86 binary at /opt/homebrew/Cellar/cffi/1.16.0_1 from the bottled version. So that's good.

But running python -m build in the source directory for python-cffi still produces the complaint about the deployment target:

building '_cffi_backend' extension
creating build/temp.macosx-14-arm64-cpython-310
creating build/temp.macosx-14-arm64-cpython-310/src
creating build/temp.macosx-14-arm64-cpython-310/src/c
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -DFFI_BUILDING=1 -I/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/ffi -I/private/var/folders/b1/0rcybhys4051l25g6yvknchh00010n/T/build-env-0s3xagif/include -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c src/c/_cffi_backend.c -o build/temp.macosx-14-arm64-cpython-310/src/c/_cffi_backend.o -iwithsysroot/usr/include/ffi
clang-6.0: error: invalid version number in 'MACOSX_DEPLOYMENT_TARGET=14'
error: command '/usr/local/clang6/bin/clang' failed with exit code 1

This may be an issue with the XCode command line tools, however, it is not fixed by reinstalling and selecting them.

[arigo]

That looks unrelated to CFFI. Do you manage to build any CPython extension module from source, or are you getting the same error? If you do, maybe you should look for help in places more generic than CFFI.

[PeterNSteinmetz]

@arigo Thanks, that is a good point. I have also started a somewhat more general question at stackoverflow - https://stackoverflow.com/questions/77734641/clang-6-0-complaint-of-invalid-version-number-in-macosx-deployment-target-14 . No answers yet.

I was able to get that exact clang6 command to produce a binary outside the build system, so it doesn't seem like a compiler problem per se.

I also note that python-cffi does not have an arm64 binaries for the current MacOS version, which suggests they are not being built presently.

That is a good idea to try and isolate the problem with a different project. Is there another Cpython extension module you would suggest I try to build from source as an example?

[PeterNSteinmetz]

So I found a hint to what might be going on in the pytables project at https://github.com/PyTables/PyTables/issues/887 and more information about where the Apple binaries are supposed to be at https://stackoverflow.com/questions/64992467/mac-clang-installation-seems-to-override-gcc-install .

It appears that some older version of homebrew or another project required installing a clang6 version at /usr/local/clang6 and having that earlier in your PATH. Thus clang was invoking an older version of the compiler which did not understand 'MACOSX_DEPLOYMENT_TARGET=14'.

I finally realized what was going on when I invoked clang6 --version and noticed it was giving me an x86 compiler with an older version. So that is the thing to check if you are having a similar problem.

Once the paths were set correctly and the correct clang was being used, I was able to build cffi from source.