search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.45k stars 13.65k forks source link

Allow pre-/post-releases in pythonRuntimeDepsCheckHook #301698

Open genevieve-me opened 5 months ago

genevieve-me commented 5 months ago

Describe the bug

In its current implementation, python-runtime-deps-check-hook.py instantiates a Requirement from packaging.requirements, which has a SpecifierSet with prereleases=False. https://packaging.pypa.io/en/stable/requirements.html

Because of this, several Python packages within nixpkgs cause this hook to fail. One example is psycopg.optional-dependencies.pool. Upstream keeps the version at "x.y.dev1" except on release tags, but psycopg-pool is built from the release tag of psycopg.

In practice, that means that building any python package with any ~= or >= version specifiers for psycopg-pool, or any other python library with a version in nixpkgs like x.y.a2, n.m.beta, etc., will fail. (I'm not sure why the packaging package behaves like this, since it seems obvious that 3.0.0.dev1 >= 2.0 is true, so I might raise an issue there).

Unfortunately, you can't override the version specifier set of a Requirement, so the hook implementation would need to be slightly refactored to use a SpecifierSet with the parameter prereleases=True: https://packaging.pypa.io/en/stable/specifiers.html.

I'm not sure if there's a better way to approach this, or if it makes more sense to generally try to avoid prerelease versions in nixpkgs, but this is currently breaking a build for me so I wanted to raise it. Thanks for your attention!

This issue was previously mentioned by @mweinelt at https://github.com/NixOS/nixpkgs/pull/270457#issuecomment-1830086703, but the implementation was kept and the package in question was updated to a non-prerelease version.

Add a :+1: reaction to issues you find important.

genevieve-me commented 5 months ago

https://github.com/pypa/packaging/issues/790

mweinelt commented 5 months ago

I'm pretty sure this behavior is in place, so that pre-releases need an explicit opt in, and won't be automatically selected as the latest and greatest version. Many packages in fact don't want to upgrade to pre-release versions.

Packages that do can usually opt in by specifying the pre-release version in the requirement specifier. That should make the comparison work.

genevieve-me commented 5 months ago

OK, not getting an alpha release by default does make sense for upstream behavior. However, I tried specifying pre-releases and still find the results a little odd, eg Requirement("foo>0.1.dev1").specifier.contains("1.0.dev1") is False.

Getting back to nixpkgs, is there a policy about packaging developmental releases? My naive reaction is that they will often fail pythonRuntimeDepsCheckHook for most packages consuming them as a dependency, so it would be nice to avoid that, but I'm sure there is a lot of context I'm missing.

For example, the exact python library that bit me was psycopg-pool, and I don't face the issue with a small patch to nixpkgs to build it from the pool-3.2.1 tag instead of the same tag as its parent psycopg. Would such a PR be welcome?