[MNT]: Slowly move towards pyproject.toml?

[oscargus]

Summary

I was considering moving parts of setup.py and setup.cfg to pyproject.toml. As I understand it, it is possible to mix and match, so whatever can be moved (I can move) will cooperate with what is left.

Since we have a setup.py with a bit of magic in it, I do no expect that I will be able to completely transition, but is it worthwhile even starting? Or do we expect that moving away from setup.py (which I guess should be the long term goal) requires more or less rewriting everything anyway, possibly changing packaging tool etc?

Proposed fix

No response

[ianthomas23]

I am in favour of this. The pybind11 work becomes much simpler if we at least replace setup_requires in setup.py with [build-system] in pyproject.toml. I was going to talk about it tomorrow's call.

FWIW Bokeh has just made this transition.

[oscargus]

I gave it a quick try to just move over the more basic things and got stuck in that setupext could not be imported from setup.py anymore. I take it that it executes from another directory or something so that it does not find setupext.py.

Copying the contents doesn't seem like the best way to solve it...

[ianthomas23]

A temporary environment is used to build a wheel which is then installed into your working environment. The temporary environment doesn't seem to have access to the full project directory.

I have had success with:

1. Removing the setup_requires section from setup.py

2. New pyproject.toml containing

   [build-system]
   build-backend = "setuptools.build_meta"
   requires = [
   "certifi>=2020.06.20",
   "numpy>=1.19",
   "setuptools_scm>=7",
   ]

3. Add the following line to setup.py just before import setupext:

   sys.path.append(str(Path(__file__).resolve().parent))

so basically botching the sys.path so that it can find setupext.py.

[oscargus]

I've seen some problems with numpy versions. My guess is that using pyproject.toml somehow uses a different version of NumPy headers than possibly installed. For example, when installing using 1.22 installed I got an error about not matching API versions, which I didn't get with a plain setup.py install, while upgrading to 1.23 (and reinstalling, so not completely sure) I didn't. Seen similar problems with another package with a much simpler install procedure as well.

Note that in both cases, the actual compilation happened in setup.py, so moving all the way may be a solution.

[tacaswell]

@oscargus See https://github.com/matplotlib/matplotlib/pull/23829/files#r968896893 for the numpy API version issue.

[QuLogic]

This was completed as part of the Meson port in #26621.