search

tttapa / py-build-cmake

Modern, PEP 517 compliant build backend for creating Python packages with extensions built using CMake.
https://pypi.org/project/py-build-cmake
MIT License
38 stars 6 forks source link

Compiling on `macos-14` (arm64) cp39 result in `Unable to determine extension suffix. Try manually setting PY_BUILD_EXT_SUFFIX.` #24

Closed laggykiller closed 3 months ago

laggykiller commented 4 months ago

When compiling a nanobind project with py-build-cmake, I encountered Unable to determine extension suffix. Try manually setting PY_BUILD_EXT_SUFFIX.. Note that my project is using QueryPythonForNanobind.cmake as provided in the (nanobind example of this project)[https://github.com/tttapa/py-build-cmake/blob/rework-0.2.0/examples/nanobind-project/cmake/QueryPythonForNanobind.cmake].

Project where I discover this problem: https://github.com/laggykiller/apngasm-python/tree/stubgen-official GitHub action log: https://github.com/laggykiller/apngasm-python/actions/runs/8062229801/job/22021550265#step:4:2210

tttapa commented 4 months ago

It seems that CMake thinks it's cross-compiling. From the logs, py-build-cmake did not enable cross-compilation, so I'm guessing that Conan enabled it. It would be useful to see the contents of /Users/runner/work/apngasm-python/apngasm-python/conan_output/armv8/conan_toolchain.cmake. My guess is that Conan considers armv8 a different target than the system's architecture and therefore enables cross-compilation. Looking at the source code, you can find the reasons why Conan might enable cross-compilation on macOS here: https://github.com/conan-io/conan/blob/656814c69f9e2f1cf1f53cfac994ad41f7476baf/conan/tools/cmake/toolchain/blocks.py#L823

Given that CMake is in cross-compilation mode, the error you got makes sense: when cross-compiling, by definition, CMake cannot query the system it's running on to get information about the target system, so the user should specify the details such as the Python extension suffix explicitly. This is by design.

However, since you did not intend to cross-compile, the main question is why CMake is in cross-compiling mode, and I think you'll find the answer in your Conan toolchain file.

tttapa commented 4 months ago

~~As a side note: the reason why it works for Python 3.8 is that SETUPTOOLS_EXT_SUFFIX was set in the environment (by Pip or cibuildwheel, I'm not sure), and this tells QueryPythonForNanobind.cmake what it needs to know to successfully cross-compile the Python module.
The same variable is not set in the Python 3.9 environment.~~

Edit: The reason why it works for Python 3.8, is that cibuildwheel uses an x86_64 version of Python 3.8, even on Apple silicon. See https://github.com/pypa/cibuildwheel/pull/1169.
Therefore, you are actually cross-compiling (the build Python is a different architecture than the target Python), and py-build-cmake handles this case correctly by setting SETUPTOOLS_EXT_SUFFIX to the appropriate value.

For Python 3.9 and up, cibuildwheel uses a native arm64 version of Python, so py-build-cmake does not (and should not) enable cross-compilation in this case.

You can see in the output of the Actions run whether py-build-cmake enabled cross-compilation or not: https://github.com/laggykiller/apngasm-python/actions/runs/8062229801

Python 3.8
py_build_cmake.config.quirks:ARCHFLAGS was specified. Automatically enabling cross-compilation for arm64 (native platform: x86_64)
Python 3.9
py_build_cmake.config.quirks:ARCHFLAGS was set, adding CMAKE_OSX_ARCHITECTURES to cmake.options (arm64, native platform: arm64)

laggykiller commented 3 months ago

Thank you for your insight, I see why it is failing now:

This arm64 macOS runner skips cp38 (Using x86_64 python) and compile >cp39 (Using arm64 python): https://github.com/laggykiller/apngasm-python/actions/runs/8074468917/job/22059804243 . Compilation was successful.

The build script I am using for the project skips running conan if conan_output is already present. Therefore if I run cp38 first, the conan_output would contain settings for crosscompiling, contaminating subsequent arm64 python builds.

I tried to force regenerate conan profile and run conan install for each build, and compilation was successful: https://github.com/laggykiller/apngasm-python/actions/runs/8074648904/job/22060296197