search

Cantera / enhancements

Repository for proposed and ongoing enhancements to Cantera
11 stars 5 forks source link

Port `setuptools` Python build to meson or scikit-build - sdist/wheel only! #205

Open bryanwweber opened 1 week ago

bryanwweber commented 1 week ago

Abstract

In https://github.com/Cantera/pypi-packages/issues/3, we discussed how to include libhdf5 support in Cantera's Python wheels. I expressed some reservations about the complexity of the existing setup and wanted to see if we can simplify that, while also giving end users the ability to configure how a wheel is built from an sdist. I will investigate meson and scikit-build-core to see if either would be an improvement over setuptools.

Out of scope:

  1. I am not considering enscons because, although it's been updated fairly recently, some of the issues on the repo suggest to me that it's not widely used or supported (for example, not supporting Python 3.12). On the other hand, both meson and scikit-build-core are extremely well supported (the former by NumPy/SciPy, the latter by a lot of the ML community).

Acceptance criteria:

  1. I've recorded in this issue the outcome of this investigation with any tradeoffs associated with each system
  2. libhdf5 support is included in the Python wheels that we build in CI
  3. End users can compile an sdist into a wheel that includes libhdf5 support from a libhdf5 library on their system

Related: #93

bryanwweber commented 1 week ago

The NumPy CI config for wheel building seems fairly simple: https://github.com/numpy/numpy/blob/main/.github/workflows/wheels.yml

NumPy heavily uses features from meson to find all the possible CPU micro-optimizations that are supported on a particular platform. See all the meson.build files they have. For our purposes, we don't need to support so many options. One of the neat things I've noticed in their config is support for OpenBLAS on Linux, Accelerate on macOS, and no BLAS on Windows. This would be a nice thing for Cantera to support as well.

bryanwweber commented 1 week ago

Right now, I'm strongly leaning towards meson. Its syntax is much closer to Python and NumPy provides a great reference. On the other hand, scikit-build-core is essentially a wrapper for CMake, which I know very little about. I'll work up a PR to use meson to build wheels and see where I can get.

bryanwweber commented 6 days ago

With a very basic meson.build, I'm not able to build the sdist due to missing features in Meson:

❯ python -m build --sdist .                                                                                                                         at 16:53:00 
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - Cython>=3.0.6
  - meson-python>=0.15.0
* Getting build dependencies for sdist...
* Building sdist...
+ meson setup /home/bryan/GitHub/cantera/build/python_sdist /home/bryan/GitHub/cantera/build/python_sdist/.mesonpy-ui1cxn2o -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/bryan/GitHub/cantera/build/python_sdist/.mesonpy-ui1cxn2o/meson-python-native-file.ini
The Meson build system
Version: 1.4.1
Source dir: /home/bryan/GitHub/cantera/build/python_sdist
Build dir: /home/bryan/GitHub/cantera/build/python_sdist/.mesonpy-ui1cxn2o
Build type: native build
Project name: Cantera
Project version: 3.1.0a2
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C++ linker for the host machine: c++ ld.bfd 2.38
Cython compiler for the host machine: cython (cython 3.0.10)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/tmp/build-env-h95vdrng/bin/python)
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Run-time dependency python found: YES 3.12
Has header "Python.h" with dependency python-3.12: YES 
Build targets in project: 0

Cantera 3.1.0a2

  User defined options
    Native files: /home/bryan/GitHub/cantera/build/python_sdist/.mesonpy-ui1cxn2o/meson-python-native-file.ini
    buildtype   : release
    b_ndebug    : if-release
    b_vscrt     : md

Found ninja-1.12.1 at /home/bryan/.local/share/mise/installs/ninja/latest/bin/ninja
+ meson dist --allow-dirty --no-tests --formats gztar
Dist currently only works with Git or Mercurial repos

ERROR Backend subprocess exited when trying to invoke build_sdist

I guess since this is being built from a subdirectory of the repo, rather than the root, it doesn't find the git directory? Anyways, this is pretty disappointing. I'll look into scikit-build-core next.

bryanwweber commented 5 days ago

After about an hour of work, I've got something to compile from Scons -> sdist -> scikit-build-core -> Cython -> cpp -> .so. It doesn't include all the pyx sources so I haven't tried installing/importing but it's further than I got with meson! Will keep pursuing this path.

From the docs, including env-var configuration should also be possible: https://scikit-build-core.readthedocs.io/en/latest/configuration.html#configuring-cmake-arguments-and-defines