Use the [project] section in pyproject.toml according to PEP-621

MartinThoma commented 3 years ago

[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have searched the documentation and believe that my question is not covered.

Feature Request

PEP621 (Storing project metadata in pyproject.toml) and PEP631 (Dependency specification in pyproject.toml based on PEP 508) define a clear way how metadata and dependencies should be stored. Both of them have the status "accepted". It would be good if poetry would use that.

For example, I've noticed that poetry init generates a pretty similar pyproject.toml, but puts the metadata in a poetry-specific section.

sinoroc commented 3 years ago

The status of PEP 621 is still "Draft". But yes, I guess it's a valid request (assuming it actually gets accepted and depending under which form). It might take a while though... the way I see it, there are still too many uncertainties

MartinThoma commented 3 years ago

Uh, interesting. My bad. However, PEP-631 is accepted. Do you see uncertainties there as well?

sinoroc commented 3 years ago

At some point, I thought there was an intention to merge the text of 631 into 621, to avoid having to refer to two documents, since 631 is rather short and unsurprising. Don't know if it will happen, or if it stays as 2 separate documents. I lost track of that conversation. And there are some more points of 621 that are still being discussed.

Now, if I put myself in the shoes of poetry:

PEP 621 might seem like a step to the side, no big changes forward or backward, but we move from non standard to standard. So probably a slight positive.
PEP 631 could definitely be seen as a step backwards.

Personally I am curious to see how poetry will handle this. There is always the dynamic escape hatch for dependencies. If I were poetry, I would probably try to offer both dependency notations (the existing poetry notation as dynamic and the new/old PEP 631 one). I had suggested a compromise hybrid notation here but it didn't gain any traction at all (probably because too complex to parse, I don't know).

MartinThoma commented 3 years ago

there was an intention to merge the text of 631 into 621

It's mentioned in the abstract of PEP-0631. However, I don't know when this will happen and where the discussion around that is.

PEP 631 could definitely be seen as a step backwards.

I'm surprised by that. Why do you think so? To me, it seems like a step towards more standardization and thus positive.

I would probably try to offer both dependency notations

Yes, I think that would be desirable just to not break a working setup for anybody. Is the poetry dependency notation more powerful or even different at all?

sinoroc commented 3 years ago

Is the poetry dependency notation more powerful or even different at all?

It's all in the discussions, particularly this one, I don't want to repeat things and take them out of context:

https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599

finswimmer commented 3 years ago

The discussion about how the dependencies should be defined within PEP-621 was quite said and frustrating for us as poetry maintainers. But I will not go into details now...

As @sdispater has said somewhere in the discussion, poetry will probably not be one of the earlier adopters of PEP-621 now PEP-631 is accepted. We have a problem here now: On the one hand, we want to support as many standards as possible and on the other, user experience is an important topic for us. And we believe that PEP-631 will be a step backwards compared to what we have now. A dilemma we have to discuss. :(

stephen-dexda commented 3 years ago

PEP 621's status is now provisionally accepted as of 8 days ago.

From the point of view of a package dev., having a build tool-independent metadata specification in pyproject.toml is useful to be able to parse metadata from the file without adding build tool-specific references elsewhere, and without duplicating information.

Use case examples:

project.scripts.*: when building container images, to determine which file to reference as to be executed.
project.requires-python: to determine which Python versions to run tests against.
project.optional-dependencies.test: to install test dependencies.

I have been using/parsing all of the above based on the PEP 621/631 drafts, and duplicating this info in pyproject.toml for now from the tool.poetry sections.

Fundamentally, project metadata is not only used by build tools, so having Poetry support (provisionally) accepted standards for these allows the developer to use the metadata for other cases without duplication, and having this support from Poetry would be fantastic :)

stephen-dexda commented 3 years ago

Additionally, PEP 621 is now referenced at https://packaging.python.org/specifications/declaring-project-metadata/.

stephen-dexda commented 3 years ago

As part of this, poetry version <rule> should bump project.version as well as/instead of tool.poetry.version, depending which is/are present.

yoursvivek commented 3 years ago

Status of PEP 621 is now: Final.

Announcment by Bret Cannon on python discussion forum.
PR on python/peps is now merged in master.

mvaled commented 3 years ago

The way I see it, poetry can start by supporting PEP 621. PEP 631 (actually PEP 508) requires some changes in the way versions are dealt with in Poetry (e.g ~= 3.6 instead of ^3.6).

stephen-dexda commented 3 years ago

PEP 621 was updated to incorporate the decision on PEP 631 vs. PEP 633; it requires PEP 508 version strings.

zubieta commented 3 years ago

Hi poetry maintainers, do you think is there a way to start supporting everything else in pep 621 while the dependency list discussion happens? I don't want to misrepresent anyone so correct me if I'm wrong, as I understood the discussion it looks like pretty much everything else (name, version, python_requires, etc) was generally accepted and shouldn't impact poetry usability.

PS. Thank you for this awesome piece of software.

eli-schwartz commented 3 years ago

If even permitting users to specify PEP compliant dependency versions is a dealbreaker (remember, people wanting poetry style versions can just specify that dependencies is a dynamic metadata key), perhaps poetry could refuse to support any pyproject.toml that doesn't specify it in dynamic?

"Error: poetry requires specifying dynamic dependencies in order to manage it differently".

Then people could use the PEP, have a PEP compliant pyproject.toml, and get some benefits of the PEP (and test out using it in poetry), even if poetry doesn't accept every possible way of following the PEP.

sinoroc commented 3 years ago

Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic escape hatch for dependencies.

To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)

stephen-dexda commented 3 years ago

The solution seems pretty straightforward to me in principle - fully/properly support PEP621 (and that includes dependency specification per PEP631), and if there is some form of dependency specification that can't be expressed in PEP631 format, then either that can be expressed in a dynamic section, or Poetry can still use its own sections (or some combination of both, e.g. use-poetry-section-for-deps = true in dynamic).

PEP621 doesn't have to support everything Poetry's config. supports for it to be supported by Poetry, it doesn't make sense to look at this issue with that assumption. It just needs to be able to sufficiently define project metadata in at least some cases for Poetry to be able to use [project] to build the project. Which it does (and probably does in most cases).

sinoroc, see https://github.com/python-poetry/poetry/issues/3332#issuecomment-735415766 in answer to your last question.

stephen-dexda commented 3 years ago

If even permitting users to specify PEP compliant dependency versions is a dealbreaker (remember, people wanting poetry style versions can just specify that dependencies is a dynamic metadata key), perhaps poetry could refuse to support any pyproject.toml that doesn't specify it in dynamic?

"Error: poetry requires specifying dynamic dependencies in order to manage it differently".

Honestly, this seems like the worst of all worlds.

eli-schwartz commented 3 years ago

Honestly, this seems like the worst of all worlds.

I'm not irrationally suggesting that poetry becomes worse as a result of the PEP because of the disagreement over version descriptions. Using dynamic is a perfectly reasonable escape hatch for people who prefer poetry's format.

Given some people do seem to think that merely offering the option to use PEP 631 is "bad", I figured a partial implementation is better than no implementation, because a partial implementation is partially useful whereas no implementation has no usefulness.

I don't actually think this is a remotely ideal situation...

zubieta commented 3 years ago

Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic escape hatch for dependencies.

To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)

@sinoroc I'm personally more interested in having the non-dependencies metadata in a single place so that several tools/scripts can reuse it without the hassle of keep it in sync. For example for a couple of projects I use poetry and towncrier:

[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "My awesome package."

[tool.towncrier]
package = "mypackage"
package_dir = "src"
filename = "NEWS.rst"
version = "0.1.0"

Like this example I have more related to documentation and publishing tools, albeit mostly for custom scripts atm.

RootLUG commented 3 years ago

Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic escape hatch for dependencies.

To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)

One of the big advantages of using PEP621 would also be that 3rd party tools would understand the dependencies. I for example have a project that is doing static analysis of a code and also dependencies (flagging outdated ones, potentially malicious, vulnerable packages, etc). If every tool has its own format of how dependencies are declared then the static analyzer must have a separate support for each of these tools which are quite hard to maintain over time.

If all the tools start migrating to use the PEP621 then that would mean also a more secure python ecosystem as the static analyzers can all use the same metadata information regardless if the project is using poetry, setuptools, flit, or anything else.

There are several other metadata info in the toml file that is very helpful for static analyzers and auditing tools which is very painful to support and maintain given all the different formats that each packaging tool has.

finswimmer commented 3 years ago

PEP621 in general is good idea and can even make poetry's live easier. @sdispater is one of the authors of this PEP.

The hard question is how the dependency specification in poetry project should look like. poetry projects doesn't have just run time dependencies, they also have dev-dependencies and more dependency groups are planned. How do we want to be able to use poetry specific flags like develop=True or allow-prereleases = true. As poetry evolves more flags will arise.

stephen-dexda commented 3 years ago

@finswimmer I answered that exact question in https://github.com/python-poetry/poetry/issues/3332#issuecomment-826324556.

finswimmer commented 3 years ago

For sure dynamic is an option. But - at least for poetry - the dependencies are the most important metadata and it would be a bit strange if poetry tries to push other packaging tool to write there dependency according to PEP631 and doesn't do this itself.

sinoroc commented 3 years ago

poetry projects doesn't have just run time dependencies, they also have dev-dependencies

On this one specific thing, I wish packaging tools would agree on some conventions here. I for example like and use the convention of having a dev extra for those development dependencies. Works amazingly well with setuptools and tox for example.

gbdlin commented 3 years ago

Is there any summary what we will lose when moving project dependencies into the PEP-621/PEP-631 specification?

I know optional dependencies/extras will be defined quite different and wouldn't allow to automatically re-use the same packet version in multiple extras groups in the same way, but there is another option for that and should already work in other build backends. Basically, you can define your dependencies like:

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]
lint = [
  "black",
  "flake8"
]
ci = [
  "beaglevote[test]",
  "beaglevote[lint]"
]

And the ci extras should include everything from the test and lint. We can support that syntax officially and document it as something recommended.

Another thing I know of is defining the source of specific packages - I understand why [project] is not supporting that, as this is not something for the packages, but for non-package projects it is at least useful.

henryiii commented 3 years ago

See https://twitter.com/HenrySchreiner3/status/1403442566109007874/retweets/with_comments 😍

For the problem of dev dependencies, there's no problem with also allowing tool.poetry as well. So what about adding a new mode in addition to the normal table mode for dev-dependencies? Like this:

[project]
name = "beaglevote"

[project.optional-dependencies]
dev = [
  "pytest < 5.0.0",
]

[tool.poetry]
dev-dependencies = "dev"

This would allow you to reuse the dependencies specified in the extras (in either "classic" mode or PEP 621 mode).

Edit: I don't think "self-referencing" works quite as expected, so I've adjusted the above to a string instead of a list ["beaglevote[dev]"]

henryiii commented 2 years ago

Update: self referencing now works in pip (21.2+, actually). Also, Flit, trampolim, and whey all support PEP 621 metadata.

gbdlin commented 2 years ago

@henryiii is the self-referencing in pip documented somewhere? Or is it just some side-effect of how pip works internally?

henryiii commented 2 years ago

https://github.com/pypa/pip/issues/10393

flying-sheep commented 2 years ago

I’m not quite sure I understand the problem here. I see three concerns here:

`allow-prereleases`

This is the only really awkward one, I think you need to specify a prerelease version here (.alphaX, .betaX, .rcX, .preX, .dev0) to enable it, so you’re unable to replace { version = '>=1.0', allow-prereleases = true }, only sth. like >=1.0.post0.dev0, which isn’t the same thing.

Caret operator

Its absence isn’t that big of a deal, you can just say x >=2.4.1, <3 instead of x = '^2.4.1', not as elegant, but … 3 keystrokes.

Abstract vs concrete dependencies

I think going forward, this 8 years old old blog post will stay relevant: https://caremad.io/posts/2013/07/setup-vs-requirement/

project metdata is for abstract dependencies (as defined in the blog post)
If people want to use concrete dependencies like git URLs, they should specify their location elsewhere

So for project.dependencies, the current specification works well, and poetry could define e.g. tool.poetry.dep-sources or something to customize where to retrieve the dependencies from.

That would also make the case more convenient where multiple environments (e.g. extras) depend on the same package

[project]
dependencies = ['floob']

[project.optional-dependencies]
test = ['floob[testing]']

[tool.poetry.dep-sources]
floob = { git = 'https://git.company.int/floob.git' }

neersighted commented 2 years ago

henryiii commented 2 years ago

That link is a 404.

pradyunsg commented 2 years ago

I'm pretty sure this has been triaged, and the labels can be updated to something like: Project Metadata, Concensus Needed, Core Changes Needed

CAM-Gerlach commented 2 years ago

Just FYI, Setuptools has added experimental support in pypa/setuptools#1688 and publicly announced it.

AFAIK, this will leave Poetry as the only major modern build backend lacking support for the standard.

EwoutH commented 2 years ago

Are there any updates on this effort?

Secrus commented 2 years ago

Are there any updates on this effort?

Right now, team is focused on delivering 1.2 release. This is gonna be addressed in future releases

henryiii commented 2 years ago

https://twitter.com/SDisPater/status/1521932867214921728

dimaqq commented 1 year ago

Ugh, so many PEPs, so confusing!

I'd love to at least be able to slap common fields (name, version, description and apparently authors) to a common [<project>] section, and give poetry free reign with dependencies. But alas!

neersighted commented 1 year ago

https://github.com/python-poetry/roadmap/issues/3#issuecomment-1186273566 is the current proposal for implementation in Poetry -- this work is likely some months out based on cycles available to the interested developers.

henryiii commented 1 year ago

You can do that according to PEP 621 - you just have to add dynamic = ["dependencies"], then Poetry can do anything it wants with dependencies in tool.poetry sections. It will not be available to tools wanting to do static analysis on the file to get the dependencies without triggering the build system, but that's the only downside.

CAM-Gerlach commented 1 year ago

Ugh, so many PEPs, so confusing!

FYI, the complete, canonical and up to date spec is in the PyPA specs section of the PyPUG, so no need to refer to a bunch of PEPs. The intention is to finish getting the canonical versions of specs migrated there to put all the key information, up to date, all in one place for spec implements; I have a PR for PEP 517/518/660, but its stalled over procedural and reviewability concerns and I haven't had the time yet to refactor and revive it.

you just have to add dynamic = ["dependencies"]

Just to be clear to anyone else reading, by "you" @henryiii means the creator of the pyproject.toml (usually the user, though they can be generated by a preprocessor tool that runs at some point before the build is invoked). You might want to mention this in your user-facing documentation, so while users can (and, when practical, should) use the project.dependencies key, as it allows deps to be statically, easily and reliably discovered by other non-poetry tooling, but also letting them know that they can still fully use Poetry's "native" dep config via the dynamic escape hatch if they prefer.

Tomperez98 commented 1 year ago

Is this still in development? The maturing project, used to build Rust libraries with a Python interface, has a conflict with Python projects manages with Poetry.

The issue comes from the fact that maturing uses a pyproject.toml compliant with PEP621

neersighted commented 1 year ago

The state has not changed from my last update in this issue; there simply has not been enough time to dedicate to this from any of the maintainers well-placed to reason about and make this change; contributions from outsiders are of course welcome, but are likely to be difficult given the complexity.

flying-sheep commented 1 year ago

@Tomperez98 you mean https://maturin.rs, right?

YodaEmbedding commented 1 year ago

One thing that seems to be missing in PEP 631 is poetry's dependency groups, e.g.

[tool.poetry.dependencies]
python = ">=3.8.0,<3.12"
typing-extensions = "^4.0.0"

[tool.poetry.group.dev.dependencies]
pytest = "^6.0.0"
pytest-mock = "*"

[tool.poetry.group.docs]
optional = true

[tool.poetry.group.docs.dependencies]
sphinx = "^4.0"
sphinx-book-theme = "^1.0.0"

[tool.poetry.group.format]
optional = true

[tool.poetry.group.format.dependencies]
black = "^23.1.0"
isort = "^5.10.1"

Usage:

poetry install --with=dev
poetry install --with=dev,docs

flying-sheep commented 1 year ago

I never quite understood why all tools add a separate way to do this when you can just use features/extras.

[project.optional-dependencies]
doc = ['sphinx']
test = ['pytest']
...  # actual optional features

and then rye sync --features=doc or [tool.hatch.envs.doc]\nfeatures = ['doc'] or poetry install -E doc

I mean sure: They’re not really features of the installed package, but they are features of the package installed in dev mode.

Secrus commented 1 year ago

I never quite understood why all tools add a separate way to do this when you can just use features/extras.

Because there is no need to pollute the extras pool for end users. For me, the existence of all those options in different tools means there is a need for such stuff.

eli-schwartz commented 1 year ago

Considering the purpose of this poetry setting is to add poetry metadata that isn't written out into the wheel metadata, I think it's unsurprising that it's not included in a spec for wheel metadata.

@YodaEmbedding why is this a problem? Surely the answer there is to continue supporting unique poetry specific features that have no overlap with PEP 621's, by not arbitrarily deleting those features as an unrelated action during the process of migrating overlapping features from the tool.poetry section to the project section?

flying-sheep commented 1 year ago

Surely the answer there is to continue supporting unique poetry specific features that have no overlap with PEP 621

that, and in parallel drafting a standard for this features that is supported by all dev tools.

I agree that “polluting the extras pool for end users” is enough reason to have it.

zmievsa commented 1 year ago

Coming here to say that this issue is very important to us and that ruff has recently become partially incompatible with poetry due to RUF200 code.

python-poetry / poetry