Difficulties in packaging

[LecrisUT]

Hi @jaraco I don't know the best place to contact you for this, but hopefully this would be a relevant place to do so.

I am trying to package some of your packages to Fedora (already packaged) and EPEL9, and most of your projects are derived from your skeleton repo and I have a few issues I would like to discuss if you could make some changes to make our lives a bit easier:

• separate testing optional dependency to be as minimal as possible. Other developer's dependencies like pytest-cov could be moved to test-cov optional dependency. Other dependencies that we are patching out: pytest-checkdocs, pytest-mypy, pytest-ruff. pytest-enabler is also disabled, but it might be good to add it eventually.
• avoid or document the reason behind the lower-bound dependencies, e.g. setuptools>=61 is unclear why this lower bound is used since PEP517 seems to have been available way before that.

This is particularly important because of the package importlib_resources that you are maintaining containing various fixes that need to be backported to lower python version and packaging to long-term distros including EPEL9.

[jaraco]

This is the right place! Thanks for laying out the issues. I plan to take a look later today.

[jaraco]

• separate testing optional dependency to be as minimal as possible. Other developer's dependencies like pytest-cov could be moved to test-cov optional dependency. Other dependencies that we are patching out: pytest-checkdocs, pytest-mypy, pytest-ruff. pytest-enabler is also disabled, but it might be good to add it eventually.

See #48 where I talk about the motivations for having these plugins in the default test suite.

I can imagine splitting out each plugin into its own extra, so running the full test suite would require installing .[test,cover,type,check,format,dist,...], but that seems like a lot of extra separation for some integrators to ignore. pytest-enabler would still need to be present or copied to each of those optional dependencies.

The only two "standard" extras are "doc" and "test", so adding new ones would be bespoke to skeleton-based projects, a convention but not a standard.

Because of the way I've implemented pytest-enabler it does mean that (unlike other projects) if you omit the plugins, they will simply not be used, so that feels like a good point for integration - the projects are lenient to removal of those dependencies, so just remove them as part of the downstream patching. And since skeleton projects are identifiable by their badge in the readme, you can use that as a signal to provide a uniform recipe for all skeleton-based projects.

So overall, I'm slightly reluctant to add the extra dimensionality of multiple optional dependencies, especially when I think about the toil this will create around applying it to hundreds of downstream projects. Let me draft the patch and see how it would look in practice and how well it merges into downstream projects.

• avoid or document the reason behind the lower-bound dependencies, e.g. setuptools>=61 is unclear why this lower bound is used since PEP517 seems to have been available way before that.

I can see the value in this. I also try to provide a reason with any pins, so you can probably look at the annotated history for any given change and see within a few commits when it was changed and hopefully why.

In the case of setuptools >= 61, I know that's there to support some of the newer features of pyproject.toml (PEP 621). That change came about when ~I~bswck applied ini2toml to the projects.

[bswck]

That change came about when I applied ini2toml to the projects.

https://github.com/jaraco/skeleton/pull/122

[LecrisUT]

I can imagine splitting out each plugin into its own extra

You can do:

[project]
name = "PROJECT"

[project.optional-dependencies]
test = [ ]
test-cov = [
  "PROJECT[test]",
]

[jaraco]

[project]
name = "PROJECT"

[project.optional-dependencies]
test = [ ]
test-cov = [
  "PROJECT[test]",
]

I'm a little reluctant to add self references, because that entails customizing that setting for each and every downstream project. If .[test] was possible, where . (or a similar construct) was meant to denote "this project", that'd be much better. Unfortunately, I don't think that's supported.

I don't think it's necessary for the ancillary dependencies. Each is fairly independent, except that pytest-enabler applies to all, so there I would want to add PROJECT[enabler] to each of the new optional dependency groups.

In #139, I explore this possibility.

[jaraco]

It also occurs to me, this concern goes away if skeleton-based projects migrate instead to the Coherent System. In that system, these dependencies are managed at the infrastructure level so would be present when using Coherent tooling, but would be undeclared so an integrator would need to replicate or tweak the Coherent tooling to install and test without them.

[bswck]

The Coherent System solves so many problems! :sweat_smile: