Eventually deprecate `setup.cfg` with automatic conversion to `pyproject.toml`

[abravalheri]

This is a follow up of #1688.

As discussed in the linked issue, the approach setuptools would like to take is to eventually use a single declarative format (pyproject.toml) instead of maintaining 2 (pyproject.toml/setup.cfg).

Originally posted by Paul Ganssle in https://github.com/pypa/setuptools/issues/1688#issue-411224984:

Add pyproject.toml as the one true way to do declarative builds, moving all setuptools configuration over there and deprecating setup.cfg (partially what is discussed in https://github.com/pypa/setuptools/issues/1160, and if we decide on that option we can move over to that issue).

Originally posted by Jason R. Coombs in https://github.com/pypa/setuptools/issues/1688#issuecomment-471212342

One way to ease the transition could be for the setup.cfg code to generate a .toml file and consume then have the pyproject.toml consumer just load both files. That way, a project seeking to transition would only need to do one build, manually merge the two .toml files, and delete setup.cfg.

Originally posted by Paul Ganssle in https://github.com/pypa/setuptools/issues/1688#issuecomment-570933225

I don't think we should warn if people use setup.cfg even when pyproject.toml is available, we should wait for pyproject.toml configuration to be stable for at least a year before actively pushing it like that.

Relevant tool: ini2toml

[thehappycheese]

Thankyou for posting this issue. As someone tinkering on the bleeding edge of packaging tools I have been so confused by the various PEPs and documentation. I thought pyproject.toml was the future but could not find anything definitive to suggest setuptools were going to deprecate setup.cfg.

[abravalheri]

Hi @thehappycheese please feel free to open a discussion if you still have doubts, either here or in https://discuss.python.org/c/packaging/14. We will try our best to help.

But meanwhile we are in no rush to deprecated setup.cfg and pyproject.toml just landed (still being evaluated).

[thehappycheese]

Thanks @abravalheri I will take a look at the forums. Yes I understand setuptools has a big community to bring along for the journey when it happens :) I think it will be a great move though if it works out

[bskinn]

@abravalheri Is that milestone correctly named (setup.py deprecation)? I still use setup.py to provide specific dynamic content to build, and I would like to be able to keep that workflow.

I suppose -- if setup.py is to be deprecated eventually, in addition to setup.cfg, are there plans to develop a replacement mechanism for providing dynamic content within a setuptools build chain?

Thanks!

[abravalheri]

Sorry, @bskinn, the correct name should be setup.cfg. I will fix it.

[mauvilsa]

Is it being considered that if setup.cfg support is removed, some features will be lost which are no possible with pyproject.toml?

I ask this because I extensively use extras_require combined with %(name)s, since it allows to define only once, dependencies used in several extras_require. Take for instance the following:

[options.extras_require]
all =
    %(retrieval)s
    %(postprocessor)s
    %(api)s
    %(validation)s
retrieval =
    %(scipy)s
    %(numpy)s
    pandas>=0.24.1
    rapidfuzz>=0.14.1
postprocessor =
    %(pagexml)s
    jsonschema>=2.6.0
api =
    %(postprocessor)s
    flask-restx>=1.0.3
validation =
    %(numpy)s
    schwifty>=2018.9.1
scipy =
    scipy>=1.3.0
numpy =
    numpy>=1.21.2
pagexml =
    pagexml-slim>=2022.4.12

Without %(name)s, for example api would need to duplicate the requirements and version specifiers of postprocessor and pagexml. Avoiding repetition is a good practice and avoids mistakes when updating the code. With pyproject.toml there is no such feature and it is unlikely to be included in the format, e.g. see https://github.com/toml-lang/toml/issues/829.

[tpvasconcelos]

AFAIK the goal is to align on self references at a higher level, such that you don't have to depend on parser specific features like configparser's interpolation. Something like:

 [options.extras_require]
 all =
-     %(retrieval)s
-     %(postprocessor)s
-     %(api)s
-     %(validation)s
+     self[retrieval]
+     self[postprocessor]
+     self[api]
+     self[validation]
 retrieval =
-     %(scipy)s
-     %(numpy)s
+     self[scipy]
+     self[numpy]
     pandas>=0.24.1
     rapidfuzz>=0.14.1
 postprocessor =
-     %(pagexml)s
+     self[pagexml]
     jsonschema>=2.6.0
 api =
-     %(postprocessor)s
+     self[postprocessor]
     flask-restx>=1.0.3
 validation =
-     %(numpy)s
+     self[numpy]
     schwifty>=2018.9.1
 scipy =
     scipy>=1.3.0
 numpy =
     numpy>=1.21.2
 pagexml =
     pagexml-slim>=2022.4.12

[mauvilsa]

AFAIK the goal is to align on self references at a higher level

Is this already supported if using pyproject.toml?

[tpvasconcelos]

Did you try it? Works for me :)

[project.optional-dependencies]
all = [
    "foo[retrieval]",
    "foo[postprocessor]",
    "foo[api]",
    "foo[validation]",
]
retrieval = [
    "foo[scipy]",
    "foo[numpy]",
    "pandas>=0.24.1",
    "rapidfuzz>=0.14.1",
]
postprocessor = [
    "foo[pagexml]",
    "jsonschema>=2.6.0",
]
api = [
    "foo[postprocessor]",
    "flask-restx>=1.0.3",
]
validation = [
    "foo[numpy]",
    "schwifty>=2018.9.1",
]
scipy = ["scipy>=1.3.0"]
numpy = ["numpy>=1.21.2"]
pagexml = ["pagexml-slim>=2022.4.12"]

[mauvilsa]

I haven't tried it yet, but I will with a real project when I have some time. Great to hear that this is possible. I did not see this in any documentation.

[cmarqu]

Here is other mentions of it: https://hynek.me/articles/python-recursive-optional-dependencies/, https://github.com/pypa/pip/issues/10393

[Eric-Arellano]

I'm excited about this change! Thank you for making Python packaging more standardized.

One thing that would help to move this along target is moving the setuptools extension for pyproject.toml out of beta (https://github.com/pypa/setuptools/issues/3683#issuecomment-1408397531). I'd like to migrate my project to pyproject.toml, but stopped because of this warning. Is it still relevant?

Support for declaring configurations not standardized by PEP 621 (i.e. the [tool.setuptools] table), is still in beta stage and might change in future releases.

[UltimateLobster]

I'm excited about this change! Thank you for making Python packaging more standardized.

One thing that would help to move this along target is moving the setuptools extension for pyproject.toml out of beta (#3683 (comment)). I'd like to migrate my project to pyproject.toml, but stopped because of this warning. Is it still relevant?

Support for declaring configurations not standardized by PEP 621 (i.e. the [tool.setuptools] table), is still in beta stage and might change in future releases.

Same here, while I don't know when setup.cfg will be deprecated, I would appreciate it if pyproject.toml would be officially out of beta. In my case I actually moved my projects to use pyproject.toml (taking the bet that major breaking changes will actually happen) and I would very much want to teach my fellow co-workers the best practices, knowing they are more stable and not likely to have major changes soon.

Is there anyway I can help with this goal?

[sinoroc]

I like having one configuration file per tool. I would be annoyed if/when I have to have the configuration of all tools in pyproject.toml. This file would potentially become quite long, and I find it would become less easy to find things. Maybe I am in the minority camp.

Maybe it would be nice if there was a convention like: pyproject.d/setuptools.toml.

[tpvasconcelos]

@sinoroc you don't have to use pyproject.toml to configure all your tools. Most tools give you the option to use their own config file (e.g. mypy.ini, ruff.toml, .isort.cfg, .flake8, .coveragerc, pytest.ini, etc...)

[jamesmyatt]

There's a trade-off between:

• Lots of sections in the pyproject.toml file. Con: it's hard to find the right section when you want to look at or modify a setting. Pro: Easy to find the file that contains the settings.
• Lots of files in the project repo root. Con: It's hard to find the right file when you want to look at or modify anything. Pro: Once you find the right file, the settings are easier to find.

For most people, they rarely look at or change the settings, so gathering them into a single file and simplifying the project repo is the much more convenient.

The idea of a pyproject.d might mitigate some of the disadvantages, but will also require a lot of design work to make it work as everyone expects for every project and would require adding more code to the tools that use pyproject.toml.

For me, the best solution would be a pre-commit hook to reorder the tables in pyproject.toml into some canonical order would seem to be the best. Like black but for pyprojroot.toml. (e.g. build-system, project, tool with nested tables ordered alphabetically).

[neutrinoceros]

For me, the best solution would be a pre-commit hook to reorder the tables in pyproject.toml into some canonical order would seem to be the best. Like black but for pyprojroot.toml. (e.g. build-system, project, tool with nested tables ordered alphabetically).

@jamesmyatt https://pyproject-fmt.readthedocs.io/en/latest/ 👀

[jamesmyatt]

@neutrinoceros Perfect. Thank you.

[sinoroc]

For the sake of the argument...

Lots of files in the project repo root. Con: It's hard to find the right file when you want to look at or modify anything.

This could be solved by something like pyproject.d/something.toml where something is what is now used in [tool.something].

But this is not going to happen. The current solution with tool tables in pyproject.toml has been widely adopted. If setuptools drops setup.cfg, it's fine, not a big deal.

[abravalheri]

Hi @sinoroc thank you very much for expressing the concerns. For the time being there it is not in the horizon of events to drop setup.cfg completely. setup.cfg has not even been deprecated yet.

Even if we eventually drop the custom parsing, chances are it will still keep working via "transpiling" it to pyproject.toml syntax for a long time, until it is safe to assume everyone moved out of it.

---

I like having one configuration file per tool.

I personally share this opinion too, but I am afraid the standardisation efforts are going in the opposite direction.

pyproject.toml started it's life as a configuration file for build frontends. The clear target for [build-system] are tools like pip, build, etc ...

Later, PEP 621 came along and added a new section ([project]) targeting build backends. It is arguable that we could get along by clustering build-frontend and build-backends in the same file, but still the separation of concerns is not complete.

Initially the spec even prevented this file from being used from other tools, but this position was modified once tools started pushing for pyproject.toml as the only means of configuration.

Maybe it would be nice if there was a convention like: pyproject.d/setuptools.toml.

That is a good idea, maybe we could make pyproject.d/<name>.toml as a replacement/complement for the tool tables... But I think before we start implementing something like that in setuptools, it would be good to have a discussion in the environment to make sure everyone agrees with this direction.