Closed evrim closed 10 months ago
Here is for pyjwt 2.8.0:
Found duplicated packages in closure for dependency 'sphinx':
sphinx 5.3.0 (/nix/store/jwkrn9p7i0r5kkdpc39c9ywfzbmvmvd0-python3.10-sphinx-5.3.0/lib/python3.10/site-packages)
sphinx 5.3.0 (/nix/store/7l4cj4g6ysi1dmz2yxl6nsq8abgiipai-python3.10-sphinx-5.3.0/lib/python3.10/site-packages)
Found duplicated packages in closure for dependency 'snowballstemmer':
snowballstemmer 2.2.0 (/nix/store/l5246ajqbmc48q792rf9xg3sqi0syfry-python3.10-snowballstemmer-2.2.0/lib/python3.10/site-packages)
snowballstemmer 2.2.0 (/nix/store/zcrf0d6by8dyrs1zmqwixrbkxpr2gpkk-python3.10-snowballstemmer-2.2.0/lib/python3.10/site-packages)
Found duplicated packages in closure for dependency 'PyStemmer':
PyStemmer 2.2.0 (/nix/store/r33m03rghvpqgz75zc8gxsgkk3w9klz2-python3.10-pystemmer-2.2.0/lib/python3.10/site-packages)
PyStemmer 2.2.0 (/nix/store/hh3pcdmca80bdvjg38s2ri9f448wvkg5-python3.10-pystemmer-2.2.0/lib/python3.10/site-packages)
typeguard-2.13.3, wrapt 1.14.1 have similar errors. Maybe there is a new way of overriding cython that I am not aware of.
Happens to me too from overriding certifi
to inject my own cert. sphinx
shows the same error you have.
Removing sphinxHook
in the packages failing to build fixes my issue, but that's not very practical to do for the volume of packages I need.
I've found another way to suppress the warnings hence the errors. Apply the following to the packages that throws conflicts in your override:
pyopenssl = super.pyopenssl.overridePythonAttrs(old: {
catchConflicts = false;
});
Obviously this doesn't fix the double packages in the closure problem. Let me see if it works.
Maybe what you're doing isn't changing pythonForBuild too. Are you using packageOverrides
like the docs show?
Please post a reproducer which can be run in the nixpkgs dir
{ pkgs ? import ./. { } }:
...
Could be because of https://github.com/NixOS/nixpkgs/commit/a3916398458f9dee7efb6cc12bc2b1d28980d38e
will have to be investigated with nix-diff
CC @kip93
The reason for me creating that PR was because of wrapt failing to cross compile due to the sphinx hook, so I tested that it both compiled and cross compiled. Plus beautiful soup since I removed the explicit workaround. So it's not the package themselves that are failing here.
From what I can see, wrapt uses sphinx-rtd-theme, which that propagates sphinx. So does the hook. So now you have 2 inputs propagating the same dependency. That's probably where the confusion is coming from?
Please post a reproducer which can be run in the nixpkgs dir
Certainly! Sorry I should have done this with the initial post.
# foo.nix
{pkgs ? import ./. {}}: let
python3 = pkgs.python3.override {
packageOverrides = final: prev: {
certifi = prev.certifi.override {
cacert = pkgs.cacert.override {
# openssl genrsa -out my_ca.crt 2048
extraCertificateFiles = [./my_ca.crt];
};
};
};
};
in
python3.pkgs.pyjwt
I bisected the failure to a3916398458f9dee7efb6cc12bc2b1d28980d38e.
Not sure how to properly fix it. For now I just reverted that commit in a fork of the nixpkgs repo and things are working again.
Well as I said that commit is just exposing an error somewhere else not the error itself, but if you're not cross compiling reverting it won't break anything for you.
Well as I said that commit is just exposing an error somewhere else not the error itself
Do you have a solution to fix this then? I'm happy to put in time to fix this on nixpkgs, I just don't know how to fix it aside from reverting the commit.
Haven't had the time to properly look into it, but my intuition tells me that we should remove sphinx as a propagated build input in sphinx themes. I feel like a theme is just resource files, nothing more; having sphinx there does not make sense to me.
I tried to remove sphinx
from sphinx-rtd-theme
, but it is not so easily done. sphinx-rtd-theme
uses the sphinx logging facilities, locale, and it has a number of places where HTML generation is dependent on sphinx version. Not impossible to remove, but it would likely break locale functionality.
Additionally sphinx-rtd-theme
uses sphinxcontrib-jquery
which uses sphinx
and has similar difficult-to-remove dependencies.
Ok, had a little time for debugging this morning, and found something interesting. While it shows a conflict, one of them does not seem to be in the closure.
$ nix why-depends -a /nix/store/n9nb3f5py32xqbgc11png2q6drdfzy5q-python3.10-pyjwt-2.8.0.drv /nix/store/z9p4x8rw1kd99lkg70rc7crw2n70jdb4-python3.10-sphinx-5.3.0.drv
'/nix/store/n9nb3f5py32xqbgc11png2q6drdfzy5q-python3.10-pyjwt-2.8.0.drv' does not depend on '/nix/store/z9p4x8rw1kd99lkg70rc7crw2n70jdb4-python3.10-sphinx-5.3.0.drv'
$ nix why-depends -a /nix/store/n9nb3f5py32xqbgc11png2q6drdfzy5q-python3.10-pyjwt-2.8.0.drv /nix/store/4nfkn104vk6q6c1h991nxn1l6dm1833v-python3.10-sphinx-5.3.0.drv
/nix/store/n9nb3f5py32xqbgc11png2q6drdfzy5q-python3.10-pyjwt-2.8.0.drv
└───/nix/store/9cs5cgli2pf71b3gpfr7n75i7mgq1npk-python3.10-sphinx-rtd-theme-1.2.0.drv
├───/nix/store/4nfkn104vk6q6c1h991nxn1l6dm1833v-python3.10-sphinx-5.3.0.drv
├───/nix/store/993c1vcna986z0fhikqqwiqd4h9z3lxq-python3.10-sphinxcontrib-jquery-3.0.0.drv
│ └───/nix/store/4nfkn104vk6q6c1h991nxn1l6dm1833v-python3.10-sphinx-5.3.0.drv
└───/nix/store/q8xsvmq2hvcj6z9klp6lbpk7fp1c4419-python3.10-readthedocs-sphinx-ext-2.2.0.drv
└───/nix/store/4nfkn104vk6q6c1h991nxn1l6dm1833v-python3.10-sphinx-5.3.0.drv
So I went ahead and tried a little something
diff --git a/pkgs/development/python-modules/pyjwt/default.nix b/pkgs/development/python-modules/pyjwt/default.nix
index cef52b754ad..4005dc840d2 100644
--- a/pkgs/development/python-modules/pyjwt/default.nix
+++ b/pkgs/development/python-modules/pyjwt/default.nix
@@ -4,9 +4,7 @@
, cryptography
, pytestCheckHook
, pythonOlder
-, sphinxHook
-, sphinx-rtd-theme
-, zope_interface
+, python
}:
buildPythonPackage rec {
@@ -31,7 +29,7 @@ buildPythonPackage rec {
"doc"
];
- nativeBuildInputs = [
+ nativeBuildInputs = with python.pythonForBuild.pkgs; [
sphinxHook
sphinx-rtd-theme
zope_interface
aaand it just works :tm:. But I'm not sure why, since unless you're cross compiling these should be the same. And indeed, they seem to be
$ nix build .#python3.pkgs.sphinx --print-out-paths
/nix/store/nscmi1vzj2m3sm99060pxw5fkl9vgd70-python3.10-sphinx-5.3.0
$ nix build .#python3.pythonForBuild.pkgs.sphinx --print-out-paths
/nix/store/nscmi1vzj2m3sm99060pxw5fkl9vgd70-python3.10-sphinx-5.3.0
Same for the hook and the theme
$ nix build .#python3.pkgs.sphinxHook --print-out-paths
/nix/store/hs8fpi0cfsqlw03q844mkmavq22z05y6-python3.10-sphinx-hook
$ nix build .#python3.pythonForBuild.pkgs.sphinxHook --print-out-paths
/nix/store/hs8fpi0cfsqlw03q844mkmavq22z05y6-python3.10-sphinx-hook
$ nix build .#python3.pkgs.sphinx-rtd-theme --print-out-paths
/nix/store/2isqzc5axx7mgwvg1lxm22f6n6xfljcm-python3.10-sphinx-rtd-theme-1.2.0
$ nix build .#python3.pythonForBuild.pkgs.sphinx-rtd-theme --print-out-paths
/nix/store/2isqzc5axx7mgwvg1lxm22f6n6xfljcm-python3.10-sphinx-rtd-theme-1.2.0
Unfortunately don't have more time today to look into this, but maybe if by the weekend noone else has chimed in I can try to find where is the phantom sphinx coming from
The issue could be that the packageOverrides aren't getting passed to pythonForBuild
Discorevee with a quick use of the reproducer and nix-diff that the extraCertificateFiles
weren't overriden
From what I can see the overrides are passed to pythonForBuild, so the why of the certificate not being there eludes me.
What I tested and seems to work is to not propagate sphinx, since most packages (with 4 exceptions) know to not depend on the propagated sphinx anyway and included either sphinx or sphinxHook explicitly. So this patch may be a better fix than the one above
diff --git a/pkgs/development/python-modules/sphinx-rtd-theme/default.nix b/pkgs/development/python-modules/sphinx-rtd-theme/default.nix
index 613f1998121..835e74bc992 100644
--- a/pkgs/development/python-modules/sphinx-rtd-theme/default.nix
+++ b/pkgs/development/python-modules/sphinx-rtd-theme/default.nix
@@ -27,10 +27,13 @@ buildPythonPackage rec {
propagatedBuildInputs = [
docutils
- sphinx
sphinxcontrib-jquery
];
+ buildInputs = [
+ sphinx
+ ];
+
nativeBuildInputs = [
pythonRelaxDepsHook
];
Tested for both native and cross compilation and they both build. Not sure how to check if the certificate is properly overriden tho.
Ok, the builds took a while but this does not seem to break anything that wasn't already broken (3 out of 4 failed to be cross compiled, but nothing related to sphinx and failed even before this change).
Does the hook even have to propagate sphinx instead of using only the binary
With the current design, hooks can't not propagate inputs (I guess it's just a matter of adding the argument, but maybe it was done this way for a reason?).
With the current design, hooks can't not propagate inputs (I guess it's just a matter of adding the argument, but maybe it was done this way for a reason?).
I mean just using substitutions
to use a absolute path to the binary
Oh, right, that makes sense. I'll go and make that change and create a PR (against staging of course) since this would be a large change and building it all on my PC would take too long.
Should be a lot better with #258223
Describe the bug
Hi there,
I've been having difficulties with the master lately.
Basically I had this custom python3.override that applies a patch to cython = super.cython.overridePythonAttrs(old: { patches = old.patches ++ [ ./some.interpreter.hack.patch ]; }); but lately unrelated packages started failing.
For example, here is a log for pyopenssl:
Any ideas? No idea why it happens or how to approach the problem.
thnx.