native pyproject.toml config

[gaborbernat]

This is a ticket to track and implement a native pyproject.toml configuration. Will update with the proposed plan and format.

Core goals:

• explicitly inheritance based (a lot of substitution at the moment is used to replicate inheritance),
• substitution able,
• reasonably familiar (e.g. share most concepts with ini format),
• succinct (some form of factors needs to have).

We'll use this opportunity to rename some keys (e.g. skipsdist should be skip_packaging, etc).

[obestwalter]

Learning from the old format, I would add:

• easy to debug (good visualization of the settings - `--showconfig`` 2.0)
• conservative error handling (err on the side of crashing early with a helpful error, rather than being too allowing for "creative" uses)

[obestwalter]

... and most importantly: a clear, versioned specification that can then evolve as we for sure won't get it right at first try :)

[obestwalter]

I think, It would be great if we could organize a sprint around this whole topic at some point this year.

[gaborbernat]

Well, most importantly we need someone try to tackle this and implement it. Personally, in my priority queue, this is quite low. I've tackled it inside https://github.com/gaborbernat/toxn/blob/master/pyproject.toml#L5 only to prove it's quite hard to get this right. I'll only have some bandwidth for this after the #1102, #998 and #338. I also want to first split up configuration module from a mega file to sub-modules.

GitHub

gaborbernat/toxn

toxn aims to automate and standardize testing in Python - next generation tox - gaborbernat/toxn

[obestwalter]

That's exactly why I think a sprint with a few more people might be a good way to get this off the ground. Maybe around a conference where a few of as will go anyway - e.g. EuroPython.

[gaborbernat]

I don't consider this a good candidate for a sprint. That creates an artificial deadline (e.g. end of sprint). Instead, we should probably propose each and every one of us our own proposal and iterate on that until we find something that can do at least what the ini does at the moment. With toxn I had to start over a few times just because halfway through I figured well this path makes feature x/y really hard to support/implement.

[ionelmc]

Why not a tox.toml? pyproject.toml is going to be a mess (just like setup.cfg was when people decided to make tools that rewrite/change stuff in it).

[obestwalter]

@gaborbernat

That creates an artificial deadline (e.g. end of sprint).

Only in your head :) A sprint would in my eyes be a good way to make progress and share state, but in no way a deadline. Especially if you explored that area already with your toxn fork, you could share what you learned and we could take it from there.

@ionelmc

Why not a tox.toml?

That would be simply the [tool.tox] part of pyproject.toml, so that comes pretty much free with it. As a user I would definitely also want the possibility to configure tox still in an extra file if it is more involved.

pyproject.toml is going to be a mess (just like setup.cfg was when people decided to make tools that rewrite/change stuff in it).

Why do you think that? I might be a bit naive because I am really not that deep in the whole packaging topic, but I am pretty hopeful that this time around we'll do things better.

[gaborbernat]

I don't think it's going to be a mess. But it definitely can be hard to maintain once many things will try to use it. E.g. a complex tox + towncrier + black config can easily make pyproject.toml 200+ lines. I think there's no reason to limit ourselves to pyproject.toml only. I would say let's have by default tox section inside pyproject.toml, but fallback to tox.toml who want to separate it out.

[ionelmc]

Why do you think that? I might be a bit naive because I am really not that deep in the whole packaging topic, but I am pretty hopeful that this time around we'll do things better.

Well ... how to put it ... I've looked at https://github.com/gaborbernat/toxn/blob/master/pyproject.toml#L5 and I thought "well I don't want to write all those ridiculously long section names, I'd already be at my wits' end after all those quotes and brackets".

The second thing is that there will inevitably be broken tools that try to modify pyproject.toml - eg: bumpversion rewrites setup.cfg (if you're naive enough to use that - I was) but with slightly different values for the other sections.

From my perspective the only benefit of toml is that tox will get more features.

To give a bit more perspective, pyproject.toml is the new hotness but also these were too the new hotness some years ago:

• setuptools - turned out to be a huge pile of buggy monkeypatching
• distribute - turned out to be a huge pile of evil monkeypatching, but with different bugs - remember those projects that messed up your python installation cause they bundled a distribute_setup.py that tried to replace installed setuptools with distribute? i do
• disutils2 - an abandoned reimplementation of distutils

What I'm getting at here: ignore the PEP518/pyproject.toml bandwagon - just think about a new configuration system for tox.

GitHub

gaborbernat/toxn

toxn aims to automate and standardize testing in Python - next generation tox - gaborbernat/toxn

[gaborbernat]

I'm a bit conflicted here now though, I don't think we can come up with anything as succinct as our current ini format. So if writing brackets and quotes is going to become a problem (which very easily could be, I found myself to be a bit awkward when I've created toxn), I don't think there's much benefit from moving off the ini format.

[ionelmc]

Hmmm, can you elaborate a bit more on this goal:

explicitly inheritance based (a lot of substitution at the moment is used to replicate inheritance)

[gaborbernat]

The idea is that now tox has a single level of implicit inheritance, e.g. toxenv base, toxenv:py overwrites. we would allow this at arbitrary level such as one could say py37 is same as py36, but override these fields. To be fair this could be added to the ini format but once factors come in becomes quite messy.

[obestwalter]

Well ... how to put it ... I've looked at gaborbernat/toxn:pyproject.toml@master#L5 and I thought "well I don't want to write all those ridiculously long section names, I'd already be at my wits' end after all those quotes and brackets".

I see - so you basically don't like the TOML format. I did not really make my mind up about it yet, but true: it is definitely more brackets/quotes heavy than our extended ini format ... but for a reason, so it might be worth it.

We'll see how things evolve, maybe pyproject.toml is indeed the next thing to prove

[obestwalter]

I'm a bit conflicted here now though, I don't think we can come up with anything as succinct as our current ini format.

I think so to. tox.ini is definitely here to stay for that very reason alone. I just think it should not grow indefinitely.

[ssbarnea]

Why TOML? I can understand that the currently ini-on-steroids used by tox does impose some challenges (like editing multiline entries with sed... ;) but if we are to use a more modern file format, why not using YAML+JINJA2 instead of adopting something so esoteric like TOML.... sounds very close to tomb :D

[gaborbernat]

@ssbarnea for me both YAML and Jinja2 feel esoteric, so I suppose it's a matter of taste. In the packaging domain toml seems to be preferred per https://www.python.org/dev/peps/pep-0518/#other-file-formats

Python.org

PEP 518 -- Specifying Minimum Build System Requirements for Python Projects

The official home of the Python Programming Language

[ssbarnea]

@gaborbernat Lets see what kind of editor support does TOML has, linters (incliding schema-verification).... so not sure if that pep was really written in a unbiased manner.

Anyway, I am really happy that the proposed format is not JSON, only XML could have being less human friendly than JSON -- not being sarcastic here.

BTW, that pep does not mention anything regarding conditional logic, something that proved to be really needed in tox and for which we used thigns like bash -c "...." to implement it. With jinja2 templating this would have being much easier to implement (including loops).

Not trying to upsell anything here but JSON and YAML do have schema verification with tens if not hundreads of schemas supported (including travis). This extension can check it in vscode.

There is a lot of value in the ability to know which options are valid or to auto complete them, making the editing of such configuration files much easier.

[gaborbernat]

Conditionals, for loops? - I feel that's more than just configuration at that point and we would enter in the world of allowing writing arbitrary code. We purposefully probably want a simple language, that keeps it low level. If you want to write for loops, conditionals, basically write your own build script I would suggest nox - https://nox.thea.codes/en/stable/ instead.

Welcome to Nox — Nox 2018.10.17 documentation

[ionelmc]

Perhaps the authors of that pep used Ansible and found that there are dozen ways to specify a string, another handful of way to format a dictionary and so on. I haven't used toml but yaml is really nasty, with horrible libraries. I have written some code to merge multiple yaml files a year ago and still regret that I haven't chosen something more simple like hjson or json5, in the addition to the chinese water torture - also known as Ansible.

[joernheissler]

Why TOML?

There are some arguments here, most of which I support: https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f What's missing are the security issues of YAML (never use yaml.load!). Aside from that, I don't have any problems using yaml, it works great for me.

I don't like TOML either (horrible syntax for nested data structures) and would have wished for some simplified YAML instead, but looks like pyproject.toml is the new standard.

JSON and YAML do have schema verification

Can't you decode the TOML and then verify the schema? I think that YAML supports everything that can be expressed in TOML (and more) so a YAML validator could be adapted to TOML easily.

[gaborbernat]

This will be possible with #1394, ETA at least September.

[ssbarnea]

@gaborbernat Maybe you can get some inspiration from https://gohugo.io/getting-started/configuration/#configuration-file which had a more flexible approach, avoiding making a decision for the config file format. In the end a config file is expected to be a nested dictionary, bit of flexibility avoided endless debates.

Regarding the conditionals question I forgot to answer. At this moment tox already does this in a various ways, for practical reasons:

• {tty:foo:bar} is a conditional
• the envname: prefix on dependencies is another hidden way to implement conditionals, so you can add a dependency only to specific environment.
• {env:FOO:default} is another kind of implementing simple conditionals based on environment values.

We should step back a little and try to see how projects are stretching the tox use and seeing how often people end-up writing bash scripts that run more complex code after realizing they waste too much time writing escapes with some bash -c "..." constructs. I am not referring to specific project, I am referring to tox.ini in general and especially across bigger projects. I am mentioning this because that refactoring would be the time to consider addressing these limitations.

For example, one thing that I like about YAML was that it was extremely simple to add a multiline string as a value, so I could easily avoid adding an extra file with a 5-10 line shell script and just embed it (indentation is the only thing needed).

A reminder to other users: you still need to control your language and follow https://github.com/tox-dev/tox/blob/master/CODE_OF_CONDUCT.md -- While I would find acceptable to hear some comments over a beer debate, try to stick to facts and avoiding attacking other entities.

[stefansjs]

so... I'm not sure about the original intention behind this ticket, but to me the most obvious reason to allow pyproject.toml has nothing to do with the superiority of toml over ini (debatable, but not the point), but the centralization of project configuration in one file. As a user, I would be very happy to avoid another configuration file if it meant I could fully specify build and test environments in one place. I believe this is the spirit of allowing extra sections in pyproject.toml in the PEP

Is there a reason in particular we can't just transliterate the current ini features into toml?

[gaborbernat]

Because the ini offers a lot of flexibility/functionality that's not easy to express within TOML, e.g. conditional factors and value substituion.

[ssbarnea]

tox usage of ini is the most complex use of the ini "format" I ever have seen. It is quite a complex DSL, even so for basic usage you do not need to read the docs. I am not saying that I love or hate it but porting it to TOML would likely end up as a disaster due to incompatible/conflicting syntax.

My impression is that tox should fix other areas before changing the configuration file format. IMHO, if the tox config should have to be rewritten, it would as well be an opportunity for the user to switch to another tool.

It may worth noting that the all tool configuration in a single file may be the wrong solution for the issue of cluttering the project root folder, especially if the tools involved are likely to have bigger configuration files with more changes to them like tox. For linters I do not see a real problem to stick them to pyproject but if you try to stick everything in, you end up with a maintenance issue, problematic backporting to stable branches and merge conflicts on bigger projects.

As I was also annoyed by the cluttering of the project root folder I started to allow configuration loading from .config folder, which is based on XDG_CONFIG_HOME -- there are several tools that can keep their config files in a subfolder. While the XDG standard refers to user home, there is nothing wrong in applying the same concepts relative to the repository root.

[gaborbernat]

I can see the appeal in having a TOML config. And we'll likely have one in tox 4. tox 4 (see rewrite branch) does a clean separation via abstraction on where the source comes from and how it is translated to python configuration objects - see https://github.com/tox-dev/tox/blob/rewrite/src/tox/config/source/__init__.py#L10. With that, we can implement a toml configuration format for tox that maybe is less succinct than the ini one, however, for simple cases is good enough. Such separation of abstractions is not present in tox 3, and introducing those is hard without breaking a lot of our plugins. Hence, why we haven't done this yet.

With that being said tox 4 is likely not to happen before the end of the year but hopeful for an RC in January. So in the meantime, this is blocked.

[ChihweiLHBird]

Is the work related to this issue still in schedule?

[jugmac00]

The current focus is on getting tox 4 ready for a final release. Just last week the final alpha was released.

This feature was scheduled for 4.1. Once we get tox 4 out, there will be quite a transition period and possibly quite some edge cases coming up which need attention.

That said - let's see what @gaborbernat has to add.

[gaborbernat]

Mostly what @jugmac00 said. The design of the tox 4 is made to allow this without too much of heavy-lift but I'd not pull myself into this when trying to make tox 4 actually come out the door. But would like to follow up on this quickly after that happens.

[ChihweiLHBird]

Thank you for the information! @jugmac00 @gaborbernat

[bagel897]

1. Could tox use the pyproject.toml to detect the supported python versions and use those?
2. If it helps, I'm making a library to support toml and ini configuration

[gaborbernat]

Thanks, but that will not help. tox configuration is complex and it's not possible to have the same config in ini and toml. E.g. substitutions and references don't work well in toml, so we need to design a config file/logic that's better suited for TOML.

[ketozhang]

@gaborbernat Since this issue has gotten as complicated as it is, it would be useful to have a writeup of the complexities you've mentioned.

[gaborbernat]

Mostly boils down to the ini logic cannot be used by the pyproject.toml file. That's because a TOML file is a type bound structure, while ini is just raw strings.

[ziima]

I'd like this to be implemented and I should have some time this summer to work on this. So I patched a prototype see #3309 to start with something. It has quite a number of limitations, see below, I'd like to confirm that I'm in the right direction first. I was hoping that in tox 4 this will be relatively easy, but this will be a long run. There is a bunch of things that hindered my progress and number of decisions to be made.

My initial idea: add tox.toml (with a tox section root) or pyproject.toml (with a tool.tox section root) to the list of sources.

Status of the prototype: Tox can handle simple tox.toml with following structure:

[tox]
min_version = '4.0'
env_list = [
    'py310',
    'py39',
    'type'
]

[testenv]
deps = 'pytest'
commands = ['pytest tests']

['testenv:type']
deps = 'mypy'
commands = ['mypy src']

I completely ditched the idea of pyproject.toml, since that would also require non-negligible work or an ugly hack.

Major limitation: Tox seems still to be heavily designed around INI structure. INI files are easy, since everything in there is just a text. Despite the fact that basic APIs in TOX 4 are designed INI-neutral (or at least at first glance), I found following problems:

• env definitions: Prototype works OK with envs defined as ['testenv:quality']. It seems to me, it would me more TOML style to have them defined as [testenv.quality]. The separator to testenv : is defined quite deep, in tox.loader.section.Section and this would require quite a deep change.
• conditionals: Tox allows conditionals almost everywhere, which is cool in INI files, but doesn't quite play well with TOML files. We should agree how are we going to define those.

Known bugs and limitations:

• TomlLoader is mostly a copy of MemoryLoader. With some patching I should be able to replace it with that one.
• TomlSource is mostly a copy od IniSource. I left it as is, until a decision on testenv naming convention for TOML files is reached.
• Some options doesn't quite work in TOML config. set_env must be a single string, I wasn't able to make extras work at all. More things may be broken. I wasn't able to completely able to make config output match respective TOML and INI files with some advanced configuration I have in one of my projects.
• Tox fails to read TOML config passed as --conf if it's name is not tox.toml. It tries to load it as INI. May be true for legacy version of pyproject.toml as well.

I plan to start with some small PRs first and move gradually towards the TOML config. A feedback would be very appreciated.

[ziima]

Ad env definitions: I don't think, it's possible to use [testenv.quality] style, since it wouldn't be possible to detect which sections define testenv and there may be collisions between option names and env names. We could stick to ['testenv:quality'], but that requires quotes in TOML. Can we use [testenv-quality] similarly to how custom definitions are defined in mypy?

[gaborbernat]

It really should be tool.tox.env instead.