Support for PEP 621

[sdispater]

Now that PEP 621 is final we will eventually need to support it.

This might be tricky since PEP 621 is not flexible enough to provide the same set of features that Poetry currently supports.

[neersighted]

Related: python-poetry/poetry#3332

[CAM-Gerlach]

Just FYI, Setuptools has added experimental support in pypa/setuptools#1688 and publicly announced it.

AFAIK, this will leave Poetry as the only major modern build backend lacking support for the standard.

[sdispater]

I tried to lay out what a PEP-621 compliant could look like. We will take the following sample project as a basis for discussion:

[tool.poetry]
name = "my-package"
version = "1.2.3"
description = "Some description."
authors = [
    "Sébastien Eustace <sebastien@eustace.io>"
]
maintainers = [
    "Sébastien Eustace <sebastien@eustace.io>"
]
license = "MIT"

readme = "README.rst"

homepage = "https://python-poetry.org"
repository = "https://github.com/python-poetry/poetry"
documentation = "https://python-poetry.org/docs"

keywords = ["packaging", "dependency", "poetry"]

classifiers = [
    "Topic :: Software Development :: Build Tools",
    "Topic :: Software Development :: Libraries :: Python Modules"
]

# Requirements
[tool.poetry.dependencies]
python = "~2.7 || ^3.6"
cleo = "^0.6"
pendulum = { git = "https://github.com/sdispater/pendulum.git", branch = "2.0" }
tomlkit = { git = "https://github.com/sdispater/tomlkit.git", rev = "3bff550", develop = false, source = "foo" }
requests = { version = "^2.18", optional = true, extras=[ "security" ] }
pathlib2 = { version = "^2.2", python = "~2.7" }

orator = { version = "^0.9", optional = true }

# File dependency
demo = { path = "../distributions/demo-0.1.0-py2.py3-none-any.whl" }

# Dir dependency with setup.py
my-package = { path = "../project_with_setup/" }

# Dir dependency with pyproject.toml
simple-project = { path = "../simple_project/" }

# Dependency with markers
functools32 = { version = "^3.2.3", markers = "python_version ~= '2.7' and sys_platform == 'win32' or python_version in '3.4 3.5'" }

# Dependency with python constraint
dataclasses = {version = "^0.7", python = ">=3.6.1,<3.7"}

[tool.poetry.extras]
db = [ "orator" ]
network = ["requests"]

[tool.poetry.group.dev.dependencies]
pytest = "~3.4"

[tool.poetry.scripts]
my-script = "my_package:main"

[tool.poetry.plugins."blogtool.parsers"]
".rst" = "some_module::SomeClass"

[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"
priority = "supplemental"

Migrating it to PEP 621 could look like this:

[project]
name = "my-package"
version = "1.2.3"
description = "Some description."
readme = "README.md"
requires-python = ">=3.6"
license = { file = "LICENSE" }
keywords = ["packaging", "dependency", "poetry"]
authors = [
    { name="Sébastien Eustace", email="<sebastien@eustace.io>" }
]
maintainers = [
    { name="Sébastien Eustace", email="<sebastien@eustace.io>" }
]

classifiers = [
    "Topic :: Software Development :: Build Tools",
    "Topic :: Software Development :: Libraries :: Python Modules"
]

# Requirements
dependencies = [
    "cleo ~=0.6",
    "pendulum @ git+https://github.com/sdispater/pendulum.git@2.0",
    "tomlkit @ git+https://github.com/sdispater/tomlkit.git@3bff550",
    "pathlib2 ~=2.2 ; python_version = '2.7'" ,
    # File dependency
    "demo @ ../distributions/demo-0.1.0-py2.py3-none-any.whl",
    # Dir dependency with setup.py
    "my-package @ ../project_with_setup/",
    # Dir dependency with pyproject.toml
    "simple-project @ ../simple_project/",
    # Dependency with markers
    "functools32 ~=3.2.3 ; python_version ~= '2.7' and sys_platform == 'win32' or python_version in '3.4 3.5'",
    # Dependency with python constraint
    "dataclasses ~=0.7 ; python_full_version >= '3.6.1' and python_full_version < '3.7'"
]

[project.optional-dependencies]
db = [
    "orator ~=0.9"
]
network = [
    "requests[security] ~=2.18"
]

[project.urls]
homepage = "https://python-poetry.org"
repository = "https://github.com/python-poetry/poetry"
documentation = "https://python-poetry.org/docs"

[project.scripts]
my-script = "my_package:main"

[project.entry-points."blogtool.parsers"]
".rst" = "some_module::SomeClass"

[tool.poetry.dependency-options]
tomlkit = { develop = true, source = "foo" }

[tool.poetry.group.dev]
dependencies= [
    "pytest ~=3.4"
]

[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"
priority = "supplemental"

Here are the noteworthy elements:

• All dependency group dependencies will be of the same format as the main dependencies declaration, i.e. PEP 508 strings. This is where the main compatibility issue will reside since it's in a location already used by Poetry.
• The usage of caret (https://python-poetry.org/docs/dependency-specification/#caret-requirements) and tilde (https://python-poetry.org/docs/dependency-specification/#tilde-requirements) requirements will be removed since they are not valid PEP 440 clauses. As such, we'll transition to compatible release clauses. Note that it might no longer be the default kind of clause in 2.0 but this will be discussed in a separate issue.
• Dependency options (like develop or source) will no longer be located in the dependency declaration but rather in a new section called dependency-options.
• Other than that each section that has an associated PEP 621 section will be relocated and adapted to acommodate any change in format.

To ease this transition, I see a few things that should be done:

1. Write a migration guide in the documentation to help people understand and make the changes needed to use this new format.
2. Implement a migrate command to alleviate the pain of this migration and let Poetry do the heavy lifting.
3. (optional) Automatically detect the old format and prompt the user to let Poetry do the migration for them.

From a pure internal standpoint, the best approach would be to have two separate "format" implementations – that can be converted from one another – that can both produce a Poetry instance that can be passed around like it is today.

[gbdlin]

That looks great, but @sdispater can you maybe add some usage of source in dependencies, so we know how it'll look like? Also, how it'll work in case of changing the build backend to another one that respects PEP 621? Can we somehow prevent dependencies from being installed from the default repository instead?

[ghost]

The usage of caret (https://python-poetry.org/docs/dependency-specification/#caret-requirements) and tilde (https://python-poetry.org/docs/dependency-specification/#tilde-requirements) requirements will be removed since they are not valid PEP 440 clauses. As such, we'll transition to compatible release clauses.

Those are not exactly equivalent to either poetry's ^ or ~ though, ideally the meaning should be preserved when migrating existing pyproject.toml files.

[neersighted]

Poetry's ^ and ~ are decomposed into pure PEP 508 clauses internally -- it is very trivial for us to make an exact conversion during an automated migration. Supporting strict PEP 508 only is the entire raison d'être of PEP 621; if Poetry makes these fields dynamic, there is no point in moving to PEP 621, as other tools will still need to invoke Poetry to understand them. In such a case, the PEP 621 syntax is strictly less ergonomic and there is no point moving 😉

[ghost]

I was not suggesting to continue supporting poetry's special syntax in the standard [project.dependencies], rather that when migrating an existing pyproject.toml it should change ^X.Y.Z to something like >= X.Y.Z, < (X+1) to preserve the meaning, because simply changing it to ~= would be an unexpected breaking change (though I believe it should only make the constraint stricter, not weaker). If I misunderstood and that's what you're already going to do then I apologize, it didn't seem like that from the example.

For newly added dependencies with poetry add I'm not opposed to it using ~= for simplicity though.

[gbdlin]

Is there any real difference between >= 1.2.3, < 1.3.0 (what Poetry's ~ means) and >= 1.2.3, == 1.2.* (what PEP 440 ~ means)?

[ghost]

~ is pretty close, but for example ~= 1.2 means >= 1.2, == 1.* (which would be ^1.2 in poetry) and ~1.2 means >= 1.2, < 1.3. So you'd have to pad it out as ~= 1.2.0.

[neersighted]

I think you misunderstood the phrasing, @nyuszika7h -- indeed, the default for poetry add was discussed in that bullet point (and that is what "transition" meant). If automated conversion is offered, we will just decompose Poetry-specific syntax into the generic forms that we use internally (as all logic in Poetry's solver/version comparison is PEP 440 based, and the special syntaxes are implemented as sugar that decomposes).

[ssbarnea]

One might want to know that setuptools-scm supports PEP-621 for quite some time and that is not the only python builder that does.

[CAM-Gerlach]

As noted above, all of the other major build backends now support PEP 621 project metadata—to correct my statement above, it is Setuptools' support for its own configuration in the pyproject.toml [tool.setuptools] table that is considered experimental (which Poetry of course has supported for a long time); its PEP 621 [project] table support has been stable since it's first major release (Setuptools 61).

[flying-sheep]

The usage of caret […] requirements will be removed

I’m looking forward to this. They should only ever be used for applications, never libraries, and it’s hard to convince newbie library maintainers to not use this feature when it’s there and advertised in the Poetry docs.

[leycec]

It's shocking this never happened. PEP 621 is – along with PEP 517, obviously – the only standard that poetry absolutely must comply with. We know that poetry must do this, because we see above that increasingly many poetry-based projects have since migrated to competing PEP 621-compliant toolchains (e.g., Hatch, PDM). poetry will never recapture those projects; they're gone.

It's been nearly two years since PEP 621 was first accepted. The window of opportunity is vanishing and may, indeed, already have closed. This is what it sounds like when build tools cry.

[edgarrmondragon]

@leycec see https://github.com/python-poetry/poetry-core/pull/567

[neersighted]

Hi all, it's well-understood that PEP 621 as the default project format is desired, and it's something the maintainers are pushing toward. However, this issue hasn't had productive discussion for some time, and has mostly devolved into me-too and "this is so simple, why don't you do it?"

I'm going to lock this issue for now, and encourage discussion on the implementation details to instead happen on the linked PR (https://github.com/python-poetry/poetry-core/pull/567). If it becomes apparent that more discussion about the overall design direction and not the specific implementation is needed, I will unlock this issue. Thanks!

[radoering]

python-poetry/poetry#9135 should be in a testable state now. (Consider it an alpha version.) Please try it out and report issues you encounter in the PR.