Update `setup.py` to remove deprecated `distutils` dependency

[AlinderS]

I encountered an issue when installing Agama recently. When running python setup.py install I get

>>> python setup.py install
Traceback (most recent call last):
  File "/home/simon/Work/Agama/setup.py", line 40, in <module>
    import os, sys, platform, subprocess, ssl, zipfile, distutils, distutils.core, distutils.dir_util, distutils.file_util
ModuleNotFoundError: No module named 'distutils'

This seems to be because the distutils standard module has been removed in Python 3.12, as announced here. PEP 632 contains the official discussion and a migration guide.

I am attempting the installation in a clean virtual environment with no other packages installed. Here is some basic information about my system, should it be relevant.

>>> platform.platform()
'Linux-6.10.5-arch1-1-x86_64-with-glibc2.40'

>>> platform.uname()[3:]
('#1 SMP PREEMPT_DYNAMIC Thu, 15 Aug 2024 00:25:30 +0000', 'x86_64', '')

>>> sys.version
'3.12.4 (main, Jun  7 2024, 06:33:07) [GCC 14.1.1 20240522]'

Thank you for your contributions to the community.

[AlinderS]

I also attempted to use pip to install, as recommended in the INSTALL file. I first installed NumPy in my virtual environment and then got

>>> pip install ./
Processing /home/simon/Work/Agama
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: agama
  Building wheel for agama (pyproject.toml) ... -
    ==== Checking supported compiler options and available libraries ====

error
  error: subprocess-exited-with-error

  × Building wheel for agama (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      warning: build_py: byte-compiling is disabled, skipping.

          ==== Checking supported compiler options and available libraries ====

      c++: error: unrecognized command-line option ‘-qno-opt-dynamic-align’
      c++ -fPIC -Wall -O2 test.cpp -o test.out
      c++ -fPIC -Wall -O2 test.cpp -o test.out -fopenmp -Werror -Wno-unknown-pragmas
      c++ -fPIC -Wall -O2 -fopenmp test.cpp -o test.out -std=c++11
      c++ -fPIC -Wall -O2 -fopenmp -std=c++11 test.cpp -o test.out -march=native
      c++ -fPIC -Wall -O2 -fopenmp -std=c++11 -march=native test.cpp -o test.out -qno-opt-dynamic-align
      c++ -fPIC -Wall -O2 -fopenmp -std=c++11 -march=native test.cpp -o test.out -Werror -Wno-missing-field-initializers
      c++ -fPIC -Wall -O2 -fopenmp -std=c++11 -march=native -Wno-missing-field-initializers test.cpp -o test.out -Werror -Wno-cast-function-type
      error: NumPy is not present - python extension cannot be compiled
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for agama
Failed to build agama
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (agama)

which seems to be what happened in #44, though the solution used there of using setup.py obviously didn't work for me.

I am using NumPy 2.1.0. My virtual environment is managed by pipenv, version 2024.0.1.

[eugvas]

Confirm, when running pip install . on a completely clean system (I am using python3.12 from homebrew on MacOS), it fails to find numpy, even if it was previously installed via pip. Note that a clean system does not contain either distutils or setuptools (the latter is a third-party package, while the former was included into standard library until version 3.12, but is no longer there). However, it appears that the first line in the setup script, import setuptools, does not trigger an ImportError exception, but rather temporarily installs setuptools, here is what it prints on my system (one has to add -v argument to pip in order to force it to print the raw output): Running command pip subprocess to install build dependencies Collecting setuptools>=40.8.0 Installing collected packages: setuptools Note also that import setuptools also makes distutils available for import, even if this is no longer a package included in the standard library. However, it appears that the setup script continues to be executed in some sort of virtual environment, which does not have access to the standard site-packages folder. So it eventually fails not because distutils is not available, but because it cannot find numpy. On my system, I was able to proceed by pip installing setuptools, then any of the three suggested installation procedures work:

• running python3.12 setup.py install in the folder containing unpacked github repository;
• running pip3.12 install -v . in the same folder;
• running pip3.12 install -v agama from anywhere.

However, for some reason, the first method puts the compiled library into ***/site-packages/agama-1.0-****.egg/agama/, rather than just site-packages/agama, so it is not importable – I had to manually move it one level up. This again seems to be a quirk of python3.12, as I have not seen this bug before – perhaps this is how they punish the folks who still attempt to use python setup.py install...

The problem raised in #44 is probably related, but has not been solved yet. I have to familiarize myself with pipenv, pyenv and the whole exciting zoo of env managers... I do not think the setup script could run without setuptools being installed, so in practice both numpy and setuptools are prerequisites. Another remaining problem is that in some cases (haven't exactly figured out when), the installer runs the whole process twice (preparing Makefile.local, then running make).

[eugvas]

update: according to https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml, "pip uses an isolated environment". one may create a file pyproject.toml with a specification of prerequisites, namely [build-system] requires = ["setuptools", "numpy"] in this case, numpy gets installed in a virtual environment together with setuptools, so that the build proceeds normally. However, in my case, this ended up using numpy 1.26 when building, but numpy 2.1 outside the temporary virtual environment, so import agama failed due to version conflict. One could also specify requirements in a [project] section of pyproject.toml, but this seems to be no different from listing them in the setup.py file itself, and does not alleviate the problem that numpy is not available in the temp environment. There is also a --no-build-isolation flag, but in my case pip (a fairly up-to-date version 24.0) does not recognize it. As mentioned earlier, the problem goes away if setuptools is already installed.

[AlinderS]

Installing setuptools with pip did not change the outcome of pip install ./ for me. But, adding simple a pyproject.toml file containing

[build-system]
requires = ["setuptools", "numpy"]

and running pip install ./ allowed the installation to proceed. It seems to have used the version of NumPy I had already installed (2.1.0) and I did not encounter any issues with version conflicts. I also had no issues with the library compiling in a strange location and needing to be moved. I can import it as expected.

Thanks a lot for your help!

[eugvas]

New sets of experiments! Setting the stage: install Python 3.12 and create an empty venv; the only package listed by pip list is pip itself. Now putting myself into the folder containing Agama source code (e.g. downloaded from github and unpacked)... pip install . => error: NumPy is not present. predictably failed; need to pip install numpy. Then running the same command produces the same error! how?? if I add a printout of sys.path at the beginning of setup.py, it turns out that the site-packages folder of my virtual environment is not on the path. Ok, perhaps adding setuptools to my venv would help? no, same error. Now, out of desperation, I also pip-installed wheel. Bang, now the setup script works fine, and the sys.path provided to the script is very different – it contains my venv path, where numpy resides. Why this behaviour depends on the installed packages, is beyond my comprehension! Now, the setup script asks some interactive questions before compilation, and one may wish to automatically answer "yes" to them (or in some cases the installation is done in an environment where the console input is not available, e.g. in a docker). In that case, there is an option --yes that can be passed to the setup script, which does exactly that. When running via pip install, the way to pass this option is very obscure: --config-settings "--build-option=--yes". Try that... and get back to the same error "NumPy is not present", because sys.path again does not contain my current environment's site-packages. How wonderful is that! finally, there is another pip option, --no-build-isolation, which brings my site-packages back into sys.path.

Now, pip developers recommend to use the pyproject.toml file to specify the installation procedure. Fine, when I create this file and add

[build-system]
requires = ["setuptools", "numpy", "wheel"]

it does temporarily install these packages during the compilation of Agama, but they are put in a temporary folder, which is removed afterwards. So one can end up having an installed Agama module, but still no Numpy. And the path to the temporary numpy distribution gets captured into Makefile.local, which means that it is impossible to recompile the already installed library should such a need arise. So for now I am not planning to switch to pyproject.toml, as long as setup.py works.

So, to summarize:

• running pip install . or pip install agama works fine if the following packages are installed: numpy, setuptools and wheel.
• if one wants or needs to run the setup script without answering questions interactively, then one needs to add the following arguments to pip install: --config-settings "--build-option=--yes" --no-build-isolation.