Allow pre-/post-releases in pythonRuntimeDepsCheckHook

[genevieve-me]

Describe the bug

In its current implementation, python-runtime-deps-check-hook.py instantiates a Requirement from packaging.requirements, which has a SpecifierSet with prereleases=False. https://packaging.pypa.io/en/stable/requirements.html

Because of this, several Python packages within nixpkgs cause this hook to fail. One example is psycopg.optional-dependencies.pool. Upstream keeps the version at "x.y.dev1" except on release tags, but psycopg-pool is built from the release tag of psycopg.

In practice, that means that building any python package with any ~= or >= version specifiers for psycopg-pool, or any other python library with a version in nixpkgs like x.y.a2, n.m.beta, etc., will fail. (I'm not sure why the packaging package behaves like this, since it seems obvious that 3.0.0.dev1 >= 2.0 is true, so I might raise an issue there).

Unfortunately, you can't override the version specifier set of a Requirement, so the hook implementation would need to be slightly refactored to use a SpecifierSet with the parameter prereleases=True: https://packaging.pypa.io/en/stable/specifiers.html.

I'm not sure if there's a better way to approach this, or if it makes more sense to generally try to avoid prerelease versions in nixpkgs, but this is currently breaking a build for me so I wanted to raise it. Thanks for your attention!

This issue was previously mentioned by @mweinelt at https://github.com/NixOS/nixpkgs/pull/270457#issuecomment-1830086703, but the implementation was kept and the package in question was updated to a non-prerelease version.

---

Add a :+1: reaction to issues you find important.

[genevieve-me]

https://github.com/pypa/packaging/issues/790

[mweinelt]

I'm pretty sure this behavior is in place, so that pre-releases need an explicit opt in, and won't be automatically selected as the latest and greatest version. Many packages in fact don't want to upgrade to pre-release versions.

Packages that do can usually opt in by specifying the pre-release version in the requirement specifier. That should make the comparison work.

[genevieve-me]

OK, not getting an alpha release by default does make sense for upstream behavior. However, I tried specifying pre-releases and still find the results a little odd, eg Requirement("foo>0.1.dev1").specifier.contains("1.0.dev1") is False.

Getting back to nixpkgs, is there a policy about packaging developmental releases? My naive reaction is that they will often fail pythonRuntimeDepsCheckHook for most packages consuming them as a dependency, so it would be nice to avoid that, but I'm sure there is a lot of context I'm missing.

For example, the exact python library that bit me was psycopg-pool, and I don't face the issue with a small patch to nixpkgs to build it from the pool-3.2.1 tag instead of the same tag as its parent psycopg. Would such a PR be welcome?

[mweinelt]

Getting back to nixpkgs, is there a policy about packaging developmental releases? My naive reaction is that they will often fail pythonRuntimeDepsCheckHook for most packages consuming them as a dependency, so it would be nice to avoid that, but I'm sure there is a lot of context I'm missing.

There is no policy against packaging pre-releases, and it can even be helpful to ease the transition between Python releases. In practice, we're not seeing much of these kinds of conflicts.

For example, the exact python library that bit me was psycopg-pool, and I don't face the issue with a small patch to nixpkgs to build it from the pool-3.2.1 tag instead of the same tag as its parent psycopg. Would such a PR be welcome?

A tagged version like that should have no pre-release tag. For sqlalchemy we just stripped it, as that was what they did themselves in their CI when building releases.

Either way, we will be allowing prereleases after the NixOS 24.11 branch-off to make the Python 3.13 transition bearable.