Migrate to modern (non-deprecated) python setuptools usage

[ejjordan]

It is currently possible to run python setup.py install, but this is deprecated.

https://packaging.python.org/en/latest/discussions/setup-py-deprecated/

It would be good to move towards using a newer setuptools backend and probably also a pyproject.toml file. This will also eliminate the need to manage deletion of install files in the src dir.

See the demo I made here https://github.com/radical-cybertools/radical.pilot/pull/3171 To be complete the rest of the scripts would need to be moved and reworked a bit.

[andre-merzky]

Running python setup.py is indeed not the recommended way to install our stack - please use pip install or python -m pip install. Those however still use setup.py internally - but that is the recommended way as documented in the link you also include.

We are working on adding a pyproject.toml file - but that will be very basic and reference the setuptools toolchain (again as recommended on the page you linked, specifically here. The main reason to keep the bulk of the deployment spec in setup.py is that it is more flexible than a toml file. Sure, one can add python snippets in the toml too - but that is mixing YAML and Python in the same file, something for which I see no real benefit for and thus would really like to avoid.

eliminate the need to manage deletion of install files in the src dir

Not really - some install code paths still place the egg subdirs in the source tree (independent of using setup.py or toml) and we want to clean those out, and we want to dynamically derive the version and store it somewhere in the deployment tree - and the in-place deployment then requires the respective cleanup. That in itself is not a problem - other python modules like versioneer (which was specifically built for that purpose) go a similar route (changing files in the source tree).

[ejjordan]

Thanks for your response. It turns out I didn't know as much about setuptools as I thought, so I appreciate your pointers.

The egg gets placed in the build tree when doing an editable install. For "regular" installs from the source tree you also get a build dir which seems to be hard coded by setuptools. It might be a good idea to manage this as well if the goal is to keep the src tree clean.

In practical use (by non-devs) none of these points matter too much because pip uses temp dirs for install purposes.

[andre-merzky]

The use of temp directories was for a long time the only option. A couple years ago, in-tree build became available but were not the default (and broken for quite a while also). By now, it actually is the default: if you remove the rm lines at the end of our setup.py , you will find the egg dir and the VERSION file within the source tree after a pip install .. I am actually glad that in-tree builds are default now: my current work tree for radical.analytics for example is >100GB right now as I collected some experiment data there and have some test sessions lying around. The old build mechanism had to copy all that to a temp dir which takes quite a bit (and much more on shared filesystems when running large scale experiments for example) - the in-tree build just copy the files that are needed for the install.

The temp dirs are still used when pulling the packages from pypi or conda, and then the file delete does not make any sense really - but it seems not worth the effort to distinguish those two cases.

[andre-merzky]

pytoml is supported now.