Discussion: support for pyproject.toml configuration

[pganssle]

Over the last few months, I've been using setup.cfg more, and I find the syntax and types and such to be super counter-intuitive. I don't find TOML to be a breeze per se, but it's at least a lot clearer to me how to specify a string, a list, etc. We also hear a lot of cries to move towards supporting pyproject.toml as the "one file" for setup configuration, in favor of setup.cfg. I'm not terribly opposed to this, but I want to open up the discussion for a few options:

1. Add pyproject.toml as the one true way to do declarative builds, moving all setuptools configuration over there and deprecating setup.cfg (partially what is discussed in #1160, and if we decide on that option we can move over to that issue).
2. Add pyproject.toml as a second supported way to do a declarative configuration, alongside setup.cfg.
3. Stay the course with setup.cfg indefinitely.

I am deeply concerned with the large amount of churn that we're introducing into the Python packaging space, so I don't want to make a lot of changes for the sake of changes, but I do think that at least having the option to use a pyproject.toml would be a real benefit, so I'm basically in favor of option 2.

I think the biggest downside of option 2 is that it means maintaining, testing and documenting three separate locations for all of our options - how to do it in setup.cfg, how to do it in pyproject.toml, and how to do it in setup.py. I think the hard part is going from 1 way to do everything to 2 ways. Our documentation is a mess anyway, but my beautiful vision for the future is that all our examples have "tabs" that allow you to toggle them between setup.cfg and setup.py. Adding a pyproject.toml in there would probably not be the worst thing.

[jaraco]

I'd prefer option 1. I imagine during the deprecation period that the setup.cfg documentation could be removed, but a permalink to the old documentation made available for projects still relying on that technique.

One way to ease the transition could be for the setup.cfg code to generate a .toml file and consume then have the pyproject.toml consumer just load both files. That way, a project seeking to transition would only need to do one build, manually merge the two .toml files, and delete setup.cfg.

My second choice would be option 3. I'd really like to avoid supporting multiple formats that do essentially the same thing.

Another consideration I'd like to mention is the proposal I only recently started fleshing out in pypa/packaging-problems#248. Such an approach would be yet another declarative format. However, this proposed approach would enable project developers to supply metadata decoratively in the canonical format (or in a format suitable for the requisite build steps). If such an approach is deemed viable and comes to fruition, the value in migrating to pyproject.toml may prove small. I suggest postponing this transition at least until that more ambitious proposal is disposed or reaches fruition.

[njsmith]

I imagine during the deprecation period that the setup.cfg documentation could be removed, but a permalink to the old documentation made available for projects still relying on that technique.

I'm pretty sure the deprecation period would be literally forever. I bet there are projects on PyPI that (a) are dead and will never see another release, (b) are still used, and (c) depend on setup.cfg to install.

I'm not opposed to adding features to setuptools and better ways to do things, but I think it's about 100x less important than not breaking stuff.

[pradyunsg]

I'm in favour of 1, with deprecated forever status (ideally keep the heading in documentation and just remove all the content, replacing it with a paragraph saying use pyproject.toml)

[qlixed]

There is an ETA for this? Or there is still some discussion pending?.I think that Option 1 is the best, but here is not stated for sure if this will be done.

[pganssle]

@qlixed There is no ETA.

I think we probably have a rough consensus for Option 1 with an indefinite deprecation period (we don't have to specify the length of the deprecation period anyway). I won't have time to do this any time soon, but I think we can accept a contribution that does.

I think we'll need to:

1. Vendor the toml library
2. Build out a tool that converts setup.cfg to pyproject.toml
3. Make it the default way that setup.cfg is processed.

I think we can say that if options are specified in both setup.cfg and pyproject.toml (under tool:setuptools), that we throw an error and refuse to build (use one or the other, not both). I don't think we should warn if people use setup.cfg even when pyproject.toml is available, we should wait for pyproject.toml configuration to be stable for at least a year before actively pushing it like that.

We'll also need to decide how to handle the little ad-hoc "types" we've implemented like file:, attr:, etc. Are we parsing these as strings (I really hate how much string parsing we do in setup.cfg as it is)? Using a dict? Is there some facility in the TOML language that would make this easy for us?

[layday]

Is there some facility in the TOML language that would make this easy for us?

Yes, TOML supports both nested and inline tables. For example, the version could be hardcoded as a string:

version = "0.0.0"

... or it could be a table specifying one of attr or file:

version = { attr = "setuptools.version" }

The same 'trick' is used by Poetry to separate dependency markers from version specifiers.

[brainwane]

In the fundable packaging improvements list I summarized this as follows (if it's inaccurate, I'd appreciate a correction, especially as a pull request):

Add support for pyproject.toml as a way to configure setuptools

setuptools does not yet allow project creators to use the new pyproject.toml standard configuration file to configure setuptools behavior. This distracts and confuses package creators, and prevents platforms and tools from depending on the presence of standard pyproject.toml metadata in packages. We'd like to implement pyproject.toml configuration support in setuptools. This requires backend development work, technical writing, and coordination and publicity work among Python users.

I'd also like to better work out what this entails and how long it would take. To dig into @pganssle's proposed sequence:

I think we'll need to:

1. Vendor the `toml` library

2. Build out a tool that converts `setup.cfg` to `pyproject.toml`

Would this live within setuptools or be part of some other codebase?

3. Make it the default way that `setup.cfg` is processed.

One subtask being: decide and implement how to deal with it if people use setup.cfg even when pyproject.toml is available.

Also: should this task include implementing a deprecation notices to output when the user's supplied a setup.cfg file, advising them to switch to pyproject.toml in the future?

So that's the backend development work. I'd add in:

4. technical writing, and coordination and publicity work among Python users:

• update all the bits of PyPUG that mention setup.cfg
• announce on Discourse & distutils-sig and deal with any questions/concerns that come up from other packaging tools maintainers
• write/update user help for https://setuptools.readthedocs.io/
• get user testing for the change (for instance, look around GitHub for issues that stem from setuptools not having this feature, and point them at the branch to test it)
• once it's landed in a released version, advertise a bit with some blog posts, Twitter threads, conference talks, and podcast appearances

What have I missed or gotten wrong? Is this mostly right?

[jaraco]

Thanks @brainwane for pushing this forward. Your outline looks good to me. And nicely done making docs and the user experience first-class considerations.

Would this live within setuptools or be part of some other codebase?

My instinct is within, as it has little value outside the context of setuptools. I would expect it to be largely uncoupled with most of the functionality in setuptools except config-related handling.

[pganssle]

One thing to note: There is an as-yet-unreleased PEP that is in its final stages that intends to standardize the form of all the build metadata in pyproject.toml. I believe that should be released for public comment soonish. (Hopefully I'm not spilling the beans too much by mentioning it here).

It's fine to start work on pyproject.toml support, but we should be aware that any ad-hoc configuration we come up with in the tool:setuptools table may become Yet Another Location for build metadata if the PEP is accepted. If someone is itching to get started on this and doesn't have access to the PEP, they can let me know and I'll see about arranging to get them a copy. Otherwise I'll link to it here when it is made public. Even before it's accepted I think coding assuming it will be accepted is reasonable; we can always rename the tables later, and actually trying to implement the thing will probably give the authors good feedback.

[KOLANICH]

If someone is itching to get started on this and doesn't have access to the PEP

Are there any deep reasons to prepare it non-publicly?

[pganssle]

Are there any deep reasons to prepare it non-publicly?

It was not my choice, but PEPs are usually prepared privately, by a single individual or a small group. This one has a decent number of authors on it (as you'll see), so it's a bit unusual in that respect, but I think it was probably a smart strategy to get an initial draft that is agreeable to a bunch of the stakeholders before bringing it forward for wider comment. It's super easy to get bogged down in random details when there are "too many cooks in the kitchen".

Honestly, it's not some secretive packaging cabal, just a project that people have been putting a lot of effort into that I happen to know about. I'm fairly certain the number of goats sacrificed was in the single digits.

[eli-schwartz]

Nice! I've been wondering for a long time why every tool inefficiently reimplements the exact same metadata verbatim in a non-introspectable manner. Good to hear this mistake is being rectified. :)

[pradyunsg]

the number of goats sacrificed was in the single digits

WAIT. Who sacrificed goats!? :o

[pradyunsg]

tool.setuptools table may become Yet Another Location for build metadata if the PEP is accepted

I mean, to an extent. The tool.setuptools will still be relevant since there's a lot of stuff that's just not performed/treated the same way (eg: C extensions, including/excluding specific files, package data etc), so there would still be setuptools specific options/flags, which it'd be nice to be able to declare in pyproject.toml.

[brainwane]

PEP 621 ("Storing project metadata in pyproject.toml") is now published in draft form and available for discussion.

[brainwane]

I talked with a few maintainers and here is a rough estimate of how much time this would take, assuming that the Python developer(s), tech writer(s), coordinator(s), etc. involved already know some things about Python packaging but haven't previously worked on the setuptools codebase.

Here's a very rough idea (always assuming people are working about 35 hrs/week, and including testing and iterating in response to code review):

1. Ramp-up time for everyone who doesn't already know the setuptools codebase: ~2 weeks
2. Vendor the toml library: ~1 week (should be fairly straightforward except for review coordination)
3. Build out a tool, to live within setuptools, that converts setup.cfg to pyproject.toml: ~4 weeks (including a few weeks of finding sample setup.cfg files, testing those, finding bugs, making design decisions about validity)
4. Make it the default way that setup.cfg is processed (including: dealing with simultaneous setup.cfg & pyproject.toml presence, and implementing deprecation notices): ~4-6 weeks

Simultaneous with a bunch of that: technical writing, and coordination and publicity work among Python users:

• update all the bits of PyPUG that mention setup.cfg: ~3 days, including review coordination
• announce on Discourse & distutils-sig and deal with any questions/concerns that come up from other packaging tools maintainers: ~3 days
• write/update user help for https://setuptools.readthedocs.io/ : ~1-2 weeks, including review coordination and finding other stuff that needs fixing
• get user testing for the change (for instance, look around GitHub, StackOverflow, the Python subreddit, etc. for people with issues that stem from setuptools not having this feature, and recruit those users at the branch to test it), and respond to those user tests: ~1 week
• once it's landed in a released version, advertise a bit with some blog posts, Twitter threads, conference talks, and podcast appearances: 2 days to 3 weeks

[KOLANICH]

Some comments:

1. Since we don't directly define console_scripts entry points anymore, I'd like to make a step further and also standardize an icon per a script. An icon is given as a relative path to a graphic file of PNG format (further other formats may be added). The action of it is following: metadata dir is given a subdir icons where icons are placed using last components of their paths as names (so they must be unique), and icons names are placed into entry points metadata (not yet standardized, I gonna probably write a pep, basically it is entry_point_name @ {"icon":"a.png"}). Different scripts can have the same icon, in this case only 1 file is kept. pip or other tools may use this info for creating shortcuts.
2. instead of dynamic I propose the following scheme: the metadata may be set by build plugins, like setuptools_scm, so instead of mentioning the name in dynamic we just mention it within the key that it is set using the plugin somehow. When a plugin is installed, it contains in its entry point metadata the list of pyproject.toml metadata keys it can (and must, if mentioned) set, so everything can be verified statically without invoking the plugin.

[eli-schwartz]

What do icons have to do with console scripts which are only relevant on the console, which doesn't have the concept of graphics?

How does this relate to consolidating existing metadata into pyproject.toml?

[KOLANICH]

What do icons have to do with console scripts which are only relevant on the console, which doesn't have the concept of graphics?

1. CLI apps may have icons. In Windows.
2. shortcuts / .desktop files to cli apps may have icons.

How does this relate to consolidating existing metadata into pyproject.toml?

In no way to consolidating, but this is also a good moment to introduce something new

[merwok]

This issue is focused on defining a new way of specifying existing information. Please create a new thread on https://discuss.python.org/c/packaging/14 to propose changes to the metadata spec. Thanks!

[nschloe]

FWIW, PEP 621 for Storing project metadata in pyproject.toml has been provisionally accepted.

[abitrolly]

@brainwane how to apply to https://github.com/psf/fundable-packaging-improvements/blob/master/FUNDABLES.md#add-support-for-pyprojecttoml-as-a-way-to-configure-setuptools ? I can start on vendoring toml and code that reads values from it for setup.cfg override. Logging hours on UpWork, working through gitcoin.co contract are all fine with me.

[di]

Hi @abitrolly, that list is projects that need funding, not projects that we already have funding for. We'd need to find funding for this first.

[berislavlopac]

FWIW, as announced in November, PEP 621 was declared Final on 1st March.

[FelixBenning]

I am a bit confused by the current state of things #1160 was closed "in favour of" this merge request and states that setup.py will not be deprecated while setup.cfg would be deprecated indefinitely. Now the documentation (e.g. Quickstart) has tabs for setup.cfg and setup.py but the default tab is setup.cfg which kind of implies that this is the recommended way of doing things and the Quickstart documentation also includes a section called: Transitioning from setup.py to setup.cfg

Lastly I wonder how a declarative format would allow things like loading the README.md from file into a variable and using that as the long description of the package, which is common I think. In other words: how are declarative configurations doing these dynamic setup logic types of things.

I assume that the above is the reason setup.py will not be deprecated but I still wonder what belongs into the pyproject.toml and setup.py then, and how they interoperate. And the current documentation only references pyproject.toml shortly in the beginning for defining the setuptools and wheels requirement.

[FelixBenning]

Lastly I wonder how a declarative format would allow things like loading the README.md from file into a variable and using that as the long description of the package, which is common I think

It appears the pyproject.toml will have this functionality included by default: https://www.python.org/dev/peps/pep-0621/#readme

[JulienPalard]

Lastly I wonder how a declarative format would allow things like loading the README.md from file into a variable and using that as the long description of the package, which is common I think. In other words: how are declarative configurations doing these dynamic setup logic types of things.

This kind of thing is common yes, and handled using attr:, file:, ... prefixes.

See for example my https://github.com/JulienPalard/oeis/ project that dropped setup.py on jan 2021 (it was previously a 2 line file: just import setuptools and setuptools.setup(), I sometime even used __import__("setuptools").setup() for the fun of oneliners).

Since PEP 621 has been accepted, the future as I personally see it, is to drop setup.cfg too in favor of pyproject.toml.

[KOLANICH]

It can be possible to create a bot searching GitHub for setup.cfgs with full metadata, converting them into PEP 621-compliant pyproject.toml and sending PRs.

[wodny]

I am a bit confused by the current state of things #1160 was closed "in favour of" this merge request and states that setup.py will not be deprecated while setup.cfg would be deprecated indefinitely. Now the documentation (e.g. Quickstart) has tabs for setup.cfg and setup.py but the default tab is setup.cfg which kind of implies that this is the recommended way of doing things and the Quickstart documentation also includes a section called: Transitioning from setup.py to setup.cfg

Documentation in various places seems to be out of sync, imprecise, confusing and sometimes it omits information if it's about PEP-517 or legacy mode.

As I understand it - the build tools environment currently undergoes both a rapid development and unification. Unification comes mainly from PEP-517, PEP-518, PEP-621 and PEP-631.

With setuptools as the build backend

When using setuptools, two modes are available:

• PEP-517: pyproject.toml + setup.cfg and optional setup.py (setuptools.build_meta)
• legacy: setup.py and optional setup.cfg (setuptools.build_meta:__legacy__).

In PEP-517 mode setup.cfg will be useful until all the things from setup.cfg can be expressed in pyproject.toml. pyproject.toml + setup.cfg without setup.py is mentioned in setuptools changelog (v40.9.0). pyproject.toml is not explicitly mentioned there but is implied by the PEP-517 mode.

In legacy mode setup.cfg gives an option to express metadata in a declarative manner with setup.py containing just the minimal code (from setuptools import setup; setup()) . For me it's more elegant than specifying static data using code.

A legacy mode with just setup.cfg but without setup.py is possible but discouraged. (setuptools provides minimal setup.py on-the-fly in both modes). See setuptools implementation.

With something else as the build backend

When not using setuptools I suppose pyproject.toml is already sufficient as the only metadata file because tools like flit or poetry support PEP-517 etc. and started using pyproject.toml instead of tool-specific config files a long time ago. setup.py is not required (but sometimes used?) and setup.cfg is just a setuptools-specific thing (probably).

Remarks

Various tools are capable of reading different things from pyproject.toml: basic metadata like author, dependencies, tool-specific settings. Note the unified [project] section as opposed to [tool.*] tool-specific sections.

Other capabilities important in this whole process but depending on a tool used:

• editable installs: not specified in PEP-517 (removed) but possible; classic for legacy mode
• reading program's version
• how dependencies are expressed

Please correct me if I'm wrong. Those are just parts of my own notes based on issues, PRs and docs of pip, setuptools, PyPA's build, flit and poetry.

[Wenzil]

@wodny In PEP-517 mode, it seems setup.py is not optional if you want to install the package in editable mode, e.g. with pip install -e .

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /Users/Wenzil/my_repo
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

I resolved it by adding a dummy setup.py file (alongside setup.cfg):

#!/usr/bin/env python
import setuptools

if __name__ == "__main__":
    setuptools.setup()

Posting in case it can help others.

[wodny]

In PEP-517 mode, it seems setup.py is not optional if you want to install the package in editable mode, e.g. with pip install -e .

I think you may be using an old version of pip (< 21.1). See the commit that changed that. What's your version of pip?

[Wenzil]

You are right!

I was on pip 20.1. I have upgraded to 21.1 and it now works as expected: setup.py is not needed anymore.

Thanks!

[abravalheri]

Hello all,

@brainwane and @pganssle previously mentioned the idea of building a tool that converts setup.cfg into pyproject.toml as one of the steps required to support PEP621.

I recently started working in a tool that might relevant here: ini2toml (it was literally created with the same objective in mind).

It is still a work in progress but I would be super happy to collaborate with setuptools!

There are a few examples that show how the translation could work out.

I had to make some assumptions because there is information/keywords in setup.cfg not covered by PEP 621, but that is pretty much provisional and can be changed according to what setuptools decide to adopt. (I also took some suggestions from this thread).

---

Thank you very much all the maintainers of setuptools! It is an amazing tool that deserves loads of love in the community

[CAM-Gerlach]

I wish I had the time in the short term to dedicate this, given my at least basic background/involvement in the packaging ecosystem, extensive technical writing experience and enthusiasm for getting this done, but maybe if we can break this up into many smaller incremental subtasks, multiple collaborators could have a shot at incrementally completing this?

As a bit of a sidenote, tomli now seems to be the gold standard for a vendored toml library as (unlike uiri/toml) its minimal, well-maintained, TOML 1.0.0-conformant, and essentially purpose-built for this role, so integrating it would be almost trivial. Pip (pypa/pip#10035 ), setuptools_scm and others in the ecosystem have already adopted it as well. So at least one aspect of this should be very straightforward.

[Ambro17]

What's the best way to help supporting pep 621 into setuptools?

[cowlinator]

@CAM-Gerlach , what kind of smaller incremental subtasks should we break this up into? What are the first steps?

[CAM-Gerlach]

@cowlinator Well, that's what I was asking :slightly_smiling_face: The high-level tasks are outlined here, but if the first steps could be broken down further into more atomic subtasks, perhaps it would be easier to get the ball rolling without committing the much larger time and effort to doing everything.

[abravalheri]

@CAM-Gerlach, @cowlinator would like to review https://github.com/pypa/setuptools/pull/2970?

Although it might be a bit dense (also because of the vendored dependencies), but I tried to break down to what would be equivalent to a few PRs (github does not offer native support to stacked PRs), and there are links and a diagram in the description showing so. (I did do a later rebase fixing some merge-conflicts but nothing too serious that invalidate my previous separation).

Also notice that the PR is adding 2 vendored dependencies, so you can skip the files in setuptools/_vendor...

[abitrolly]

github does not offer native support to stacked PRs

Is there a forge that supports stacked PRs or MRs?

[abravalheri]

github does not offer native support to stacked PRs

Is there a forge that supports stacked PRs or MRs?

@abitrolly according to this article launchpad seems to support it (if I understood your question).

[CAM-Gerlach]

Wow, thanks for the work, @abravalheri ! Unfortunately, I'm far less knowledgeable about Setuptools' internals than you, let alone a core developer or maintainer, so I'm afraid I'm not much more help on the review, but good on you for taking that on! However, I can potentially help with some of the technical writing work, since that's closer to my expertise, though I'm already championing at least one major packaging PEP and helping out with several others, never mind many other tasks.

[abravalheri]

If anyone would like to help testing the feature (to see if anything breaks in existing packages), I posted some information on the Python Discourse.

[ssbarnea]

So PEP-631 is not implemented by setuptools yet... :(

[abravalheri]

Hi @ssbarnea, we are planning to merge the experimental branch very soon. Meanwhile, I am just making sure we don't break existing projects. If you want to try it out, please check the instructions in the Python Discourse. Any feedback is appreciated.

[ssbarnea]

Super. I recently discovered that dependabot supports looking at requirements from pyproject.toml but not setup,cfg and I doubt they will add them (bug is more than two years old). So for the moment I am bit stuck and unable to automate bumping of test dependencies.

[ofek]

dependabot supports looking at requirements from pyproject.toml but not setup,cfg

I think you meant the opposite: https://github.com/dependabot/dependabot-core/issues/3290

[CAM-Gerlach]

I thought it only supported requirements.txt and some custom thing with poetry; are you sure it supports setup.cfg? I've haven't ever seen any Dependabot PRs for the latter across the many projects I maintain that use both, just for GH actions versions.

[abravalheri]

The v61.*.* series of setuptools includes some initial support for pyproject.toml in an experimental fashion.

The experimental aspect of it allow us to think about how we would like the [tool.setuptools] table to look like (e.g. #3191). Currently it is mostly reflecting everything what already exists in setup.cfg, but it is pretty much open for debate.

The recently improved auto-discovery should allow the simple projects to completely skip the [tool.setuptools] table.

Note that the existing implementation started by taking approach 2, as discussed in the first post. However the plan is to eventually deprecate setup.cfg (and therefore adopt what is described in approach 1). But that will come with time.

I will open a new issue to track the deprecation.

P.S.: There is already an initial implementation of how to generate pyproject.toml from setup.cfg in ini2toml.

[EpicWink]

The v61.*.* series of setuptools includes some initial support for pyproject.toml in an experimental fashion.

you say "some" support, but I don't see what isn't supported

[abravalheri]

Hi @EpicWink, from the top of my head:

• [ ] Not all aspects of PEP 621 are completely implemented during the transition phase. Specifically, the "strictness" is relaxed to avoid breaking existing packages that were previously relying on the fact that no tool in the "build chain" was validating pyproject.toml. Ref: https://github.com/pypa/cibuildwheel/issues/1064
• [ ] The aspects of setuptools configuration not covered by PEP 621 are currently implemented via [tool.setuptools]. The final shape of [tool.setuptools] and which exact fields it will support still need to be debated and might change.
• Configurations that are currently not supported in setup.cfg are also not supported in pyproject.toml (e.g. ext_modules). In the long term we might decide to support them or not.
• setup.cfg supports even more configurations, e.g. tag_build, tag_date, etc. These need even more debate than [tool.setuptools]
• People are already starting to ask about PEP 639 (even before approval). This is currently not implemented either.