Open twhitehead opened 5 years ago
Yep, this is a known issue, one which bothers me as well. I think we should pull the creation of the package set out of the interpreters.
As a workaround, keep your overrides
as an attribute, something like pythonOverrides
. The next overlay can then use composeExtensions
.
Thanks for the suggestions and links.
If I'm understanding where you are going with your comment, you are saying if the interpreter package pulled in the package set instead of producing it, you could then override the package set in the standard way with overlays.
The interpreter would automatically pick up changes because it would be closed over the final package set. You could suck the final interpreter in the package set if you just re-exported it for compatibility without creating any loops.
Cheers! -Tyson
Exactly that!
if the interpreter package pulled in the package set instead of producing it
But, having the overrides
is enough:
overlay_a.nix
:
self: super: {
pythonOverrides = selfPython: superPython: {
py.overridePythonAttrs (oldAttrs: {
pname = "foo";
});
};
python36 = python36.pkgs.override { packageOverrides = pythonOverrides; };
}
overlay_b.nix
:
self: super: {
pythonOverrides = lib.composeExtensions (selfPython: superPython: {
pytest.overridePythonAttrs (oldAttrs: {
pname = "foobar";
});
}) super.pythonOverrides;
}
(not tested)
Thanks. I implemented that and it works well. :+1:
Unless you would like to leave it open, feel free to close this issue.
For a more standard-nixos version of said workaround, here's more or less what I'm using:
python-override-setup.nix
{ config, lib, pkgs, ... }:
{
# Setup for https://github.com/NixOS/nixpkgs/issues/44426 python
# overrides not being composable...
nixpkgs.overlays = lib.mkBefore [
(self: super: let
pyNames = [
"python27" "python34" "python35" "python36" "python37"
"pypy"
];
overriddenPython = name: [
{ inherit name; value = super.${name}.override { packageOverrides = self.pythonOverrides; }; }
{ name = "${name}Packages"; value = super.recurseIntoAttrs self.${name}.pkgs; }
];
overriddenPythons = builtins.concatLists (map overriddenPython pyNames);
in {
pythonOverrides = pyself: pysuper: {};
# The below is just a wrapper for clarity of intent, use like:
# pythonOverrides = buildPythonOverrides (pyself: pysuper: { ... # overrides }) super.pythonOverrides;
buildPythonOverrides = newOverrides: currentOverrides: super.lib.composeExtensions newOverrides currentOverrides;
} // listToAttrs overriddenPythons)
];
}
some-module.nix
{ config, lib, pkgs, ...}:
{
nixpkgs.overlays = [
(self: super: {
pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
some-package = "definition from some-package.nix";
}) super.pythonOverrides;
})
];
}
another-module.nix
{ config, lib, pkgs, ...}:
{
nixpkgs.overlays = [
(self: super: {
pythonOverrides = super.buildPythonOverrides (pyself: pysuper: {
another-package = "definition from another-package.nix";
}) super.pythonOverrides;
})
];
}
mkBefore
is to ensure that this is independent of the default order of expression evaluation.This should be solved with https://github.com/NixOS/nixpkgs/pull/54266
$ code='with import ./. { overlays = [
(self: super: {
pythonPackages.one = "definition from 1.nix";
})
(self: super: {
pythonPackages.two = "definition from 2.nix";
})
]; }'
$ nix-instantiate --eval -E "$code; pythonPackages.one"
"definition from 1.nix"
$ nix-instantiate --eval -E "$code; pythonPackages.two"
"definition from 2.nix"
I'm not quite sure about this, but .override
can be used with a function which takes the previous argument set: .override (a: { stuff = doSomething a.oldStuff})
so I think in theory one could write a function that overrides the previous arguments with a recursive merge? It's been a while but that sounds close to what I did here: https://github.com/deliciouslytyped/nix-ghidra-wip/blob/43b2207729db83f84c0812ff9a054907d5ddc222/packages.nix#L6
Actually this PR of mine solves this issue: https://github.com/NixOS/nixpkgs/pull/67422
It allows overrides of the form
python3.pkgs.overrideScope' (self: super: {
# ...
})
Without discarding previous changes.
And this with very little code changed.
if the interpreter package pulled in the package set instead of producing it
But, having the
overrides
is enough:
overlay_a.nix
:self: super: { pythonOverrides = selfPython: superPython: { py.overridePythonAttrs (oldAttrs: { pname = "foo"; }); }; python36 = python36.pkgs.override { packageOverrides = pythonOverrides; }; }
overlay_b.nix
:self: super: { pythonOverrides = lib.composeExtensions (selfPython: superPython: { pytest.overridePythonAttrs (oldAttrs: { pname = "foobar"; }); }) super.pythonOverrides; }
(not tested)
Thanks, with this i got it working. Though, it needed a few modifications to work. Here the corrected version in case anyone needs it:
overlay_a.nix
:
self: super: {
pythonOverrides = selfPython: superPython: {
py = superPython.py.overridePythonAttrs (oldAttrs: {
pname = "foo";
});
};
}
overlay_b.nix
:
self: super: rec {
pythonOverrides = self.lib.composeExtensions super.pythonOverrides (selfPython: superPython: {
pytest = superPython.pytest.overridePythonAttrs (oldAttrs: {
pname = "foobar";
});
});
python36 = super.python36.override { packageOverrides = pythonOverrides; };
}
I forgot about this thread and I haven't read carefully enough to tell if this helps here, but maybe it's relevant; https://discourse.nixos.org/t/makeextensibleasoverlay/7116
I'm now using the following function for mach-nix to merge an arbitrary list of pythonOverrides:
mergeOverrides = with pkgs.lib; overrides:
if length overrides == 0
then a: b: {} # return dummy overrides
else
if length overrides == 1
then elemAt overrides 0
else
let
last = head ( reverseList overrides );
rest = reverseList (tail ( reverseList overrides ));
in
composeExtensions (mergeOverrides rest) last;
@DavHau mergeOverrides = lib.foldr lib.composeExtensions (self: super: { })
Maybe we should put that in lib (or not) :p
@deliciouslytyped that was proposed in https://github.com/NixOS/nixpkgs/issues/33258
I marked this as stale due to inactivity. → More info
What should I do when defining overlays for nix flake workflow? If I follow the instruction at https://discourse.nixos.org/t/makeextensibleasoverlay/7116/5, then I can make overlays in my Python flake projects recomposable, but what if:
I marked this as stale due to inactivity. → More info
This is still very much a problem to be solved.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/weird-overlay-behavior-with-python-packages/14469/7
To save others the substantial amount of trouble I had to deal with trying to get the various suggestions in this thread to work, if you're using mach-nix, the easiest way to deal with overrides is to avoid this entirely and instead use its built-in override methodology: https://github.com/DavHau/mach-nix/blob/master/examples.md#simplified-overrides-_-argument
I marked this as stale due to inactivity. → More info
Not stale.
I've just encoutered this issue when trying to override the broken pyopenssl
package on aarch64-darwin
for the awscli2
package.
As you can see here
let
py = python3.override {
packageOverrides = self: super: {
awscrt = super.awscrt.overridePythonAttrs (oldAttrs: rec {
version = "0.13.11";
src = self.fetchPypi {
inherit (oldAttrs) pname;
inherit version;
sha256 = "sha256-Yx3I3RD57Nx6Cvm4moc5zmMbdsHeYiMghDfbQUor38E=";
};
});
};
};
It creates its own override of python3 meaning any of my attempts to mark pyopenssl
is not broken and skip the test phase of twisted
fail because the overrides are not composable.
Would the awscli2
derivation need to be adjusted to make use of the composeExtensions
method above?
Right now I've worked around it by forking the derivation entirely and then overlaying that instead. It really sucks that this is outside of the usually pleasant overlay system.
If you change the awscli2
definition to use overrideScope
instead it can now be properly overriden from an overlay.
# awscli2.nix
let
pypkgs = python3.pkgs.overrideScope
(self: prev: {
awscrt = prev.awscrt.overridePythonAttrs (oldAttrs: rec {
version = "0.13.11";
src = self.fetchPypi {
inherit (oldAttrs) pname;
inherit version;
sha256 = "sha256-Yx3I3RD57Nx6Cvm4moc5zmMbdsHeYiMghDfbQUor38E=";
};
});
});
in
# overlay.nix
self: prev: {
python3 = prev.python3 // {
pkgs = prev.python3.pkgs.overrideScope (self: prev: {
pyopenssl = prev.pyopenssl.overrideAttrs (_: { meta.broken = false; });
twisted = prev.twisted.overridePythonAttrs (_: { doCheck = false; });
});
};
}
I don't currently see any examples in nixpkgs
of using overrideScope with python packages, but it looks like it works.
This seems like a definite usability hurdle. Three different overrides required to set one parameter on two Python packages.
:+1: on the overrideScope
solution. Here's how I'm using it.
I have two functions that each return sets of python packages.
modifiedPackagesOverlay
takes existing packages and changes properties; its final
and prev
are sets of python3Packages
.newPackagesOverlay
adds new packages; here final
and prev
correspond to top level packages, and python dependencies have to be specified via python3Packages
.The result is an overlay, exported as part of a flake. The python3
that results from applying the overlay can be overridden in a similar manner.
{
overlays.python = final: prev:
let
pythonPkgsScope = prev.python3.pkgs.overrideScope (prev.lib.composeManyExtensions [
(_: _: newPackages final prev)
]);
in
{
python3 = prev.python3 // {
pkgs = pythonPkgsScope;
packageOverrides = lib.warn ''
`python3.packageOverrides` does not compose;
instead, manually replace the `pkgs` attr of `python3` with `python3.pkgs.overrideScope` applied to the overrides.
''
prev.python3.packageOverrides;
};
python3Packages = pythonPkgsScope;
};
}
This overlay also works if applied after one which uses pythonOverrides
, but not if applied before.
[Edit: turns out setting python3Packages = final.python3.pkgs
now breaks MacOS evaluation… I'm not sure why, but this does work.]
The nixos module system could probably be used to solve this problem nicely.
If you change the
awscli2
definition to useoverrideScope
instead it can now be properly overriden from an overlay.# awscli2.nix let pypkgs = python3.pkgs.overrideScope (self: prev: { awscrt = prev.awscrt.overridePythonAttrs (oldAttrs: rec { version = "0.13.11"; src = self.fetchPypi { inherit (oldAttrs) pname; inherit version; sha256 = "sha256-Yx3I3RD57Nx6Cvm4moc5zmMbdsHeYiMghDfbQUor38E="; }; }); }); in
# overlay.nix self: prev: { python3 = prev.python3 // { pkgs = prev.python3.pkgs.overrideScope (self: prev: { pyopenssl = prev.pyopenssl.overrideAttrs (_: { meta.broken = false; }); twisted = prev.twisted.overridePythonAttrs (_: { doCheck = false; }); }); }; }
I don't currently see any examples in
nixpkgs
of using overrideScope with python packages, but it looks like it works.
This doesn't seem to work for me. I have the following overlay applied to nixpkgs
final: prev: {
python3 = prev.python3 // {
pkgs = prev.python3.pkgs.overrideScope (python-final: python-prev: {
solidpython2 = python-final.callPackage ./solidpython2.nix { };
});
};
}
where solidpython2.nix
is somethig like
{ buildPythonPackage, fetchPypi }:
buildPythonPackage rec {
............
}
and then I have this
{
environment.systemPackages = with pkgs; [
blah
blah
(python3.withPackages (ps: with ps; [
blah
blah
solidpython2
]))
];
}
which ends in error: undefined variable 'solidpython2'
.
@WizardUli It looks like withPackages
doesn't pick up the overriden python3.pkgs
, instead it keeps referring to pythonPackages
from the let
block:
(although I don't understand how passthruFun
works and how it would be updated to pick up an overriden pkgs
)
I've had success with this kind of usage:
(python3.withPackages (_: with python3.pkgs; [
solidpython2
]))
I've had success with this kind of usage:
(python3.withPackages (_: with python3.pkgs; [ solidpython2 ]))
That's exactly what I've ended up using.
You can just use the python.buildEnv
function which withPackages
uses under the hood.
By the way, there is now pythonPackagesExtensions
attribute (added in #91850) that contains a list of overlays automatically applied to the Python packages set, and as a bonus it works for all Python versions simultaneously.
Issue description
@FRidh the
packageOverrides
function argument approach to overriding package used in the base python function (here is the 2.7 one) doesn't compose in overlays. Specifically, if you follow the directions in the nixpkgs manual only the last declared override winds up being applied.I haven't actually thought of a solution to this yet, other than determining that it is pretty much impossible to work around as you can only get a hold of a closed version of the previous package set via
python.pkgs
. Perhaps looking into the haskell infrastructure might give some directions (haven't looked at it in a while, but the fixpoint stuff looks like it came from there)?Steps to reproduce
Put the following into
~/.config/nixpkgs/overlays/1.nix
and this into
~/.config/nixpkgs/overlays/2.nix
Ideally both of these would get applied. This is not the case though as you can see from the following
If you move the
2.nix
file aside then you see the override from1.nix
but not2.nix