Open MartinThoma opened 3 years ago
The status of PEP 621 is still "Draft". But yes, I guess it's a valid request (assuming it actually gets accepted and depending under which form). It might take a while though... the way I see it, there are still too many uncertainties
Uh, interesting. My bad. However, PEP-631 is accepted. Do you see uncertainties there as well?
At some point, I thought there was an intention to merge the text of 631 into 621, to avoid having to refer to two documents, since 631 is rather short and unsurprising. Don't know if it will happen, or if it stays as 2 separate documents. I lost track of that conversation. And there are some more points of 621 that are still being discussed.
Now, if I put myself in the shoes of poetry:
Personally I am curious to see how poetry will handle this. There is always the dynamic escape hatch for dependencies. If I were poetry, I would probably try to offer both dependency notations (the existing poetry notation as dynamic and the new/old PEP 631 one). I had suggested a compromise hybrid notation here but it didn't gain any traction at all (probably because too complex to parse, I don't know).
there was an intention to merge the text of 631 into 621
It's mentioned in the abstract of PEP-0631. However, I don't know when this will happen and where the discussion around that is.
PEP 631 could definitely be seen as a step backwards.
I'm surprised by that. Why do you think so? To me, it seems like a step towards more standardization and thus positive.
I would probably try to offer both dependency notations
Yes, I think that would be desirable just to not break a working setup for anybody. Is the poetry dependency notation more powerful or even different at all?
Is the poetry dependency notation more powerful or even different at all?
It's all in the discussions, particularly this one, I don't want to repeat things and take them out of context:
The discussion about how the dependencies should be defined within PEP-621 was quite said and frustrating for us as poetry maintainers. But I will not go into details now...
As @sdispater has said somewhere in the discussion, poetry will probably not be one of the earlier adopters of PEP-621 now PEP-631 is accepted. We have a problem here now: On the one hand, we want to support as many standards as possible and on the other, user experience is an important topic for us. And we believe that PEP-631 will be a step backwards compared to what we have now. A dilemma we have to discuss. :(
PEP 621's status is now provisionally accepted as of 8 days ago.
From the point of view of a package dev., having a build tool-independent metadata specification in pyproject.toml is useful to be able to parse metadata from the file without adding build tool-specific references elsewhere, and without duplicating information.
Use case examples:
project.scripts.*
: when building container images, to determine which file to reference as to be executed.project.requires-python
: to determine which Python versions to run tests against.project.optional-dependencies.test
: to install test dependencies.I have been using/parsing all of the above based on the PEP 621/631 drafts, and duplicating this info in pyproject.toml for now from the tool.poetry
sections.
Fundamentally, project metadata is not only used by build tools, so having Poetry support (provisionally) accepted standards for these allows the developer to use the metadata for other cases without duplication, and having this support from Poetry would be fantastic :)
Additionally, PEP 621 is now referenced at https://packaging.python.org/specifications/declaring-project-metadata/.
As part of this, poetry version <rule>
should bump project.version
as well as/instead of tool.poetry.version
, depending which is/are present.
Status of PEP 621 is now: Final.
The way I see it, poetry can start by supporting PEP 621. PEP 631 (actually PEP 508) requires some changes in the way versions are dealt with in Poetry (e.g ~= 3.6
instead of ^3.6
).
PEP 621 was updated to incorporate the decision on PEP 631 vs. PEP 633; it requires PEP 508 version strings.
Hi poetry maintainers, do you think is there a way to start supporting everything else in pep 621 while the dependency list discussion happens? I don't want to misrepresent anyone so correct me if I'm wrong, as I understood the discussion it looks like pretty much everything else (name, version, python_requires, etc) was generally accepted and shouldn't impact poetry usability.
PS. Thank you for this awesome piece of software.
If even permitting users to specify PEP compliant dependency versions is a dealbreaker (remember, people wanting poetry style versions can just specify that dependencies is a dynamic
metadata key), perhaps poetry could refuse to support any pyproject.toml that doesn't specify it in dynamic?
"Error: poetry requires specifying dynamic dependencies in order to manage it differently".
Then people could use the PEP, have a PEP compliant pyproject.toml, and get some benefits of the PEP (and test out using it in poetry), even if poetry doesn't accept every possible way of following the PEP.
Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic
escape hatch for dependencies.
To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)
The solution seems pretty straightforward to me in principle - fully/properly support PEP621 (and that includes dependency specification per PEP631), and if there is some form of dependency specification that can't be expressed in PEP631 format, then either that can be expressed in a dynamic
section, or Poetry can still use its own sections (or some combination of both, e.g. use-poetry-section-for-deps = true
in dynamic
).
PEP621 doesn't have to support everything Poetry's config. supports for it to be supported by Poetry, it doesn't make sense to look at this issue with that assumption. It just needs to be able to sufficiently define project metadata in at least some cases for Poetry to be able to use [project]
to build the project. Which it does (and probably does in most cases).
sinoroc, see https://github.com/python-poetry/poetry/issues/3332#issuecomment-735415766 in answer to your last question.
If even permitting users to specify PEP compliant dependency versions is a dealbreaker (remember, people wanting poetry style versions can just specify that dependencies is a
dynamic
metadata key), perhaps poetry could refuse to support any pyproject.toml that doesn't specify it in dynamic?"Error: poetry requires specifying dynamic dependencies in order to manage it differently".
Honestly, this seems like the worst of all worlds.
Honestly, this seems like the worst of all worlds.
I'm not irrationally suggesting that poetry becomes worse as a result of the PEP because of the disagreement over version descriptions. Using dynamic
is a perfectly reasonable escape hatch for people who prefer poetry's format.
Given some people do seem to think that merely offering the option to use PEP 631 is "bad", I figured a partial implementation is better than no implementation, because a partial implementation is partially useful whereas no implementation has no usefulness.
I don't actually think this is a remotely ideal situation...
Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the
dynamic
escape hatch for dependencies.To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)
@sinoroc I'm personally more interested in having the non-dependencies metadata in a single place so that several tools/scripts can reuse it without the hassle of keep it in sync. For example for a couple of projects I use poetry and towncrier:
[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "My awesome package."
[tool.towncrier]
package = "mypackage"
package_dir = "src"
filename = "NEWS.rst"
version = "0.1.0"
Like this example I have more related to documentation and publishing tools, albeit mostly for custom scripts atm.
Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the
dynamic
escape hatch for dependencies.To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)
One of the big advantages of using PEP621 would also be that 3rd party tools would understand the dependencies. I for example have a project that is doing static analysis of a code and also dependencies (flagging outdated ones, potentially malicious, vulnerable packages, etc). If every tool has its own format of how dependencies are declared then the static analyzer must have a separate support for each of these tools which are quite hard to maintain over time.
If all the tools start migrating to use the PEP621 then that would mean also a more secure python ecosystem as the static analyzers can all use the same metadata information regardless if the project is using poetry, setuptools, flit, or anything else.
There are several other metadata info in the toml file that is very helpful for static analyzers and auditing tools which is very painful to support and maintain given all the different formats that each packaging tool has.
PEP621 in general is good idea and can even make poetry's live easier. @sdispater is one of the authors of this PEP.
The hard question is how the dependency specification in poetry project should look like. poetry projects doesn't have just run time dependencies, they also have dev-dependencies and more dependency groups are planned. How do we want to be able to use poetry specific flags like develop=True
or allow-prereleases = true
. As poetry evolves more flags will arise.
@finswimmer I answered that exact question in https://github.com/python-poetry/poetry/issues/3332#issuecomment-826324556.
For sure dynamic
is an option. But - at least for poetry - the dependencies are the most important metadata and it would be a bit strange if poetry tries to push other packaging tool to write there dependency according to PEP631 and doesn't do this itself.
poetry projects doesn't have just run time dependencies, they also have dev-dependencies
On this one specific thing, I wish packaging tools would agree on some conventions here. I for example like and use the convention of having a dev
extra for those development dependencies. Works amazingly well with setuptools
and tox
for example.
Is there any summary what we will lose when moving project dependencies into the PEP-621
/PEP-631
specification?
I know optional dependencies/extras will be defined quite different and wouldn't allow to automatically re-use the same packet version in multiple extras groups in the same way, but there is another option for that and should already work in other build backends. Basically, you can define your dependencies like:
[project.optional-dependencies]
test = [
"pytest < 5.0.0",
"pytest-cov[all]"
]
lint = [
"black",
"flake8"
]
ci = [
"beaglevote[test]",
"beaglevote[lint]"
]
And the ci
extras should include everything from the test
and lint
. We can support that syntax officially and document it as something recommended.
Another thing I know of is defining the source of specific packages - I understand why [project]
is not supporting that, as this is not something for the packages, but for non-package projects it is at least useful.
See https://twitter.com/HenrySchreiner3/status/1403442566109007874/retweets/with_comments đ
For the problem of dev dependencies, there's no problem with also allowing tool.poetry
as well. So what about adding a new mode in addition to the normal table mode for dev-dependencies? Like this:
[project]
name = "beaglevote"
[project.optional-dependencies]
dev = [
"pytest < 5.0.0",
]
[tool.poetry]
dev-dependencies = "dev"
This would allow you to reuse the dependencies specified in the extras (in either "classic" mode or PEP 621 mode).
Edit: I don't think "self-referencing" works quite as expected, so I've adjusted the above to a string instead of a list ["beaglevote[dev]"]
Update: self referencing now works in pip (21.2+, actually). Also, Flit, trampolim, and whey all support PEP 621 metadata.
@henryiii is the self-referencing in pip documented somewhere? Or is it just some side-effect of how pip works internally?
Iâm not quite sure I understand the problem here. I see three concerns here:
allow-prereleases
This is the only really awkward one, I think you need to specify a prerelease version here (.alphaX, .betaX, .rcX, .preX, .dev0) to enable it, so youâre unable to replace { version = '>=1.0', allow-prereleases = true }
, only sth. like >=1.0.post0.dev0
, which isnât the same thing.
Its absence isnât that big of a deal, you can just say x >=2.4.1, <3
instead of x = '^2.4.1'
, not as elegant, but ⊠3 keystrokes.
I think going forward, this 8 years old old blog post will stay relevant: https://caremad.io/posts/2013/07/setup-vs-requirement/
So for project.dependencies
, the current specification works well, and poetry could define e.g. tool.poetry.dep-sources
or something to customize where to retrieve the dependencies from.
That would also make the case more convenient where multiple environments (e.g. extras) depend on the same package
[project]
dependencies = ['floob']
[project.optional-dependencies]
test = ['floob[testing]']
[tool.poetry.dep-sources]
floob = { git = 'https://git.company.int/floob.git' }
That link is a 404.
I'm pretty sure this has been triaged, and the labels can be updated to something like: Project Metadata, Concensus Needed, Core Changes Needed
Just FYI, Setuptools has added experimental support in pypa/setuptools#1688 and publicly announced it.
AFAIK, this will leave Poetry as the only major modern build backend lacking support for the standard.
Are there any updates on this effort?
Are there any updates on this effort?
Right now, team is focused on delivering 1.2 release. This is gonna be addressed in future releases
Ugh, so many PEPs, so confusing!
I'd love to at least be able to slap common fields (name
, version
, description
and apparently authors
) to a common [<project>]
section, and give poetry free reign with dependencies. But alas!
https://github.com/python-poetry/roadmap/issues/3#issuecomment-1186273566 is the current proposal for implementation in Poetry -- this work is likely some months out based on cycles available to the interested developers.
You can do that according to PEP 621 - you just have to add dynamic = ["dependencies"]
, then Poetry can do anything it wants with dependencies in tool.poetry
sections. It will not be available to tools wanting to do static analysis on the file to get the dependencies without triggering the build system, but that's the only downside.
Ugh, so many PEPs, so confusing!
FYI, the complete, canonical and up to date spec is in the PyPA specs section of the PyPUG, so no need to refer to a bunch of PEPs. The intention is to finish getting the canonical versions of specs migrated there to put all the key information, up to date, all in one place for spec implements; I have a PR for PEP 517/518/660, but its stalled over procedural and reviewability concerns and I haven't had the time yet to refactor and revive it.
you just have to add dynamic = ["dependencies"]
Just to be clear to anyone else reading, by "you" @henryiii means the creator of the pyproject.toml
(usually the user, though they can be generated by a preprocessor tool that runs at some point before the build is invoked). You might want to mention this in your user-facing documentation, so while users can (and, when practical, should) use the project.dependencies
key, as it allows deps to be statically, easily and reliably discovered by other non-poetry tooling, but also letting them know that they can still fully use Poetry's "native" dep config via the dynamic
escape hatch if they prefer.
Is this still in development? The maturing project, used to build Rust libraries with a Python interface, has a conflict with Python projects manages with Poetry.
The issue comes from the fact that maturing uses a pyproject.toml
compliant with PEP621
The state has not changed from my last update in this issue; there simply has not been enough time to dedicate to this from any of the maintainers well-placed to reason about and make this change; contributions from outsiders are of course welcome, but are likely to be difficult given the complexity.
@Tomperez98 you mean https://maturin.rs, right?
One thing that seems to be missing in PEP 631 is poetry's dependency groups, e.g.
[tool.poetry.dependencies]
python = ">=3.8.0,<3.12"
typing-extensions = "^4.0.0"
[tool.poetry.group.dev.dependencies]
pytest = "^6.0.0"
pytest-mock = "*"
[tool.poetry.group.docs]
optional = true
[tool.poetry.group.docs.dependencies]
sphinx = "^4.0"
sphinx-book-theme = "^1.0.0"
[tool.poetry.group.format]
optional = true
[tool.poetry.group.format.dependencies]
black = "^23.1.0"
isort = "^5.10.1"
Usage:
poetry install --with=dev
poetry install --with=dev,docs
I never quite understood why all tools add a separate way to do this when you can just use features/extras.
[project.optional-dependencies]
doc = ['sphinx']
test = ['pytest']
... # actual optional features
and then rye sync --features=doc
or [tool.hatch.envs.doc]\nfeatures = ['doc']
or poetry install -E doc
I mean sure: Theyâre not really features of the installed package, but they are features of the package installed in dev mode.
I never quite understood why all tools add a separate way to do this when you can just use features/extras.
Because there is no need to pollute the extras pool for end users. For me, the existence of all those options in different tools means there is a need for such stuff.
Considering the purpose of this poetry setting is to add poetry metadata that isn't written out into the wheel metadata, I think it's unsurprising that it's not included in a spec for wheel metadata.
@YodaEmbedding why is this a problem? Surely the answer there is to continue supporting unique poetry specific features that have no overlap with PEP 621's, by not arbitrarily deleting those features as an unrelated action during the process of migrating overlapping features from the tool.poetry section to the project section?
Surely the answer there is to continue supporting unique poetry specific features that have no overlap with PEP 621
that, and in parallel drafting a standard for this features that is supported by all dev tools.
I agree that âpolluting the extras pool for end usersâ is enough reason to have it.
Feature Request
PEP621 (Storing project metadata in pyproject.toml) and PEP631 (Dependency specification in pyproject.toml based on PEP 508) define a clear way how metadata and dependencies should be stored. Both of them have the status "accepted". It would be good if poetry would use that.
For example, I've noticed that
poetry init
generates a pretty similarpyproject.toml
, but puts the metadata in a poetry-specific section.