search

PySCeS / pysces

The official PySCeS project source code repository.
https://pysces.github.io
Other
34 stars 10 forks source link

Implementing the `meson-python` build backend #94

Closed jmrohwer closed 6 months ago

jmrohwer commented 8 months ago

This PR implements meson-python, a PEP 517 compliant Python build backend that is suitable for building C and Fortran extension modules by implementing the meson build system. There is no more setup.py but project build options and metadata are now in pyproject.toml.

These changes are substantial, thus a version bump to 1.2.0 is proposed.

Current status (prior to this PR)

  1. PySCeS is built with the scikit-build system making use of a setup.py file. Under the hood, scikit-build makes use of cmake to build extension modules, and in our case these (Fortran) modules nleq2 and pitcon are compiled with f2py -c. The flow is thus: scikit-build calls cmake calls f2py -c.

  2. The compilation via f2py -c occurs via numpy.distutils (Python <=3.11) or meson (Python >=3.12) due to the deprecation of distutils and numpy.distutils, and thus differs between Python versions.

Motivation

  1. Python 3.12 removed distutils and as a consequence numpy removed np.distutils. This was the motivation for originally moving the build to scikit-build in PySCeS release 1.0.3 (Sep 2022) and move away from a direct np.distutils implementation.
  2. However, the current scikit-build implementation is kludgy:
    • f2py -c calls meson via cmake but since meson is used for the build, it could be called directly and the whole cmake infrastructure is actually not needed.
    • f2py -c uses different backends depending on Python version, which increases complexity.
  3. Maintainer burden will be reduced by only having one build backend.
  4. Compiling Fortran on Windows is tricky. np.distutils used to have a hack to combine MSVC (for compiling C) with gfortran (for compiling Fortran). This is however no longer supported from Python 3.12 onwards.
  5. Actually, f2py -c should not be used at all in the context of build systems such as scikit-build or meson-python, but should only be used as a transpiler to generate *.c sources which are then integrated into regular CMake or meson targets for the build. See https://github.com/numpy/numpy/issues/22211#issuecomment-1707155843
  6. The RTools toolchain has been demonstrated to work, and is used by Numpy and Scipy as well on their CI builds. It has the additional advantage (in contrast to mingw32) of preferring build of statically linked libraries, which gets rid of linking issues.
  7. Numpy and Scipy have moved their build systems to meson/meson-python.

What has changed?

  1. From a regular user perspective, not much. Installation is still via pip or conda.
  2. From a developer perspective: there is no longer a setup.py. Metadata has been migrated to pyproject.toml. Build settings are spread between pyproject.toml and various meson.build files. The build is straightforward using one of: 
    $ pip wheel -w dist .
$ python -m build .

    as long as the relevant compiler toolchain is installed (gcc and gfortran on Linux and macOS, RTools on Windows). Installation can be done with

    • pip install . or
    • pip install --no-build-isolation -e . (for an editable install). Refer to INSTALL.md
  3. Numpy version >=1.26 is required for the build, as older versions of f2py (which is distributed with numpy) are not compatible. At runtime, any numpy version >=1.23 is supported, older versions don't work due to ABI incompatibility.
  4. Binaries (wheels) are distributed for Python versions 3.9-3.12. Python 3.8 support is dropped as the latest numpy and scipy versions also no longer support it.
  5. Documentation has been updated to reflect these changes.
jmrohwer commented 8 months ago

@bgoli Any chance of a review before Christmas or are you already on leave?

jmrohwer commented 7 months ago

@bgoli Have you had a chance to look at this? I would like to cut a new release.

jmrohwer commented 6 months ago

@bgoli I'd really like to cut a release soon. We need the new version for a course we're teaching. Have you had a chance to look at this?

bgoli commented 6 months ago

Sorry about the delay I'm looking at it now, should be finished tomorrow.

On Mon, Jan 15, 2024, 11:14 Johann Rohwer @.***> wrote:

@bgoli https://github.com/bgoli Have you had a chance to look at this? I would like to cut a new release.

— Reply to this email directly, view it on GitHub https://github.com/PySCeS/pysces/pull/94#issuecomment-1891798365, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGHUELGXH4TXY67O43YCV3YOT6PZAVCNFSM6AAAAABAWD5SGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJRG44TQMZWGU . You are receiving this because you were mentioned.Message ID: @.***>