[FR] Draft support for PEP 735 Dependency Groups when building dists

sirosen commented 6 days ago

What's the problem this feature will solve?

PEP 735 Dependency Groups are a proposed draft specification for embedding non-extra dependency lists into pyproject.toml. Many of the use-cases it targets do not require support from build backends. However, one case in particular is predicated on support from backends to include dependency groups in package dependencies: exposure of package dependencies without the package itself

I would like to begin discussing implementation of this feature, explicitly as a draft until the PEP is accepted.

Describe the solution you'd like

I would like to contribute this feature to setuptools, but I have not made contributions to the project in the past.

My expectation is that:

setuptools would either
- a) import a variant of the DependencyGroupResolver I wrote along with relevant unit tests, OR
- b) depend on dependency-groups to implement resolution (following this path, I would try to move dependency-groups to pypa management and release a 1.0.0 version)
PEP 517 build hooks would be updated to support dependency group includes in package lists

Alternative Solutions

No response

Additional context

I am the author of PEP 735, and have provided the dependency-groups package as a sample implementation. I have also made a similar feature request against pip, independently, but for the pip context, I believe that a fully vendored / in-tree implementation is necessary.

dependency-groups has no dependencies of its own except for packaging.

Code of Conduct

[X] I agree to follow the PSF Code of Conduct

abravalheri commented 6 days ago

Hi @sirosen, thank you very much for starting the discussion about this.

I am sorry I did not have the time to go through the PEP yet. I will try to do that as soon as I can (but that may take a few weeks - so that is why I am providing a very quick comment below).

In principle, my personal opinion (not necessarily shared by other maintainers) is that I would like to avoid PEPs that require build backends taking extra dependencies (vendored or not) at least not until https://github.com/pypa/packaging-problems/issues/342#issuecomment-2286648465 is fully solved by the community (Python packaging as a whole probably needs to take a step back and clean-up the technical debt before moving ahead).

Can this feature be implemented without build backends (i.e. could backends simply ignore the new tables)? Since the package itself is not installed (and therefore no need for building), wouldn't this be a installer or workflow-tool, or frontend concern?

(I will probably have to update these views and questions after I have a proper read at the PEP. So I apologise in advance if these comments have already been addressed).

sirosen commented 5 days ago

No worries about taking time to read the PEP! But I hope we can make some initial progress on this thread in the meantime.

In principle, my personal opinion (not necessarily shared by other maintainers) is that I would like to avoid PEPs that require build backends taking extra dependencies (vendored or not) at least not until pypa/packaging-problems#342 (comment) is fully solved by the community (Python packaging as a whole probably needs to take a step back and clean-up the technical debt before moving ahead).

Understood. I'm already aware of pip's unique challenges around vendoring dependencies, but wasn't sure about setuptools. That said, the PEP is intentionally very simple to implement -- we need not think of the existing implementations as art to copy so much as examples.

The reference implementation for the PEP is very short, as is the optimized version I linked. I'd be more than happy to work on a setuptools-internal implementation, tailored to fit as appropriate. We're taking about a few hundred lines of code for the feature and the tests, so it's small but not shockingly so.

Can this feature be implemented without build backends (i.e. could backends simply ignore the new tables)? Since the package itself is not installed (and therefore no need for building), wouldn't this be a installer or workflow-tool, or frontend concern?

The feature has utility in both contexts. Many of the use cases focus on installers and environment managers, but the spec also allows for the following data:

[dependency-groups] 
typecheck = ["mypy", "types-redis", {include-group = "runtime"}]
runtime = ["redis", "flask"]

[project]
dependencies = [{include-group = "runtime"}]

Initially the spec did not touch the [project] table, but I eventually became convinced that it has to in order to address the pip --only-deps user story, and related cases like the above scenario.

The PEP obviously has a lot more to say on this topic, but I hope that helps give the flavor of it.

I've reached out only to setuptools and pip so far. I was thinking my next stops would be tox, maybe nox, and then some of the other build backends like hatch and flit. pdm and poetry will have more complex ux decisions to make, so I think it's best to let them decide how to proceed. The uv team is looking at the draft now, and I'm hoping for some feedback soon.

pypa / setuptools