Open rrbutani opened 1 year ago
Potential Solutions
There has got to be a simpler way of doing all of this. The fact that splicing breaks randomly in weird ways and hardly anybody understands it is a major contributor to people hating on cross compilation. There has got to be a simpler way to do all of this.
Potential Solutions
4. Reconsider whether or not splicing was a good idea
There has got to be a simpler way of doing all of this. The fact that splicing breaks randomly in weird ways and hardly anybody understands it is a major contributor to people hating on cross compilation. There has got to be a simpler way to do all of this.
https://github.com/NixOS/nixpkgs/issues/204303#issuecomment-1403662186
3: Discourage using package sets like
python3.pkgs
"directly"
I think about 8 years ago I introduced this and it has been a clear mistake. Since then this pattern was copied throughout. We should indeed go to pythonPackages
and also move the helpers such as buildEnv
and withPackages
into the package set, like haskell already had before.
@FRidh what are the costs of reverting it?
@FRidh what are the costs of reverting it?
.overrideScope
)python3Packages
throughout Basically people willing to put in that effort.
"Willing" also includes having some theoretical knowledge?
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/frustrations-about-splicing/49607/1
I ran into this while trying to get LLVM to eval under cross-compilation as part of #194634.
Apologies if this has already been addressed elsewhere; I know @Artturin has been working on fixes and cleanup for splicing recently but I didn't see this specific issue brought up in any of the existing issues and PRs.
Problem
As detailed in https://github.com/NixOS/nixpkgs/pull/194634/commits/888f966869d66b85042dc0971d1fc046578bf6f9, packages that are in package sets within derivations (i.e. the
psutil
package withinpython3.pkgs
wherepython3
is a derivation) do not get spliced.This can be observed via
__splicedPackages
(I know this is meant to be an implementation detail but it is the attrset that top-levelcallPackage
draws from and it's handy for debugging):Note that accessing the python package set via
python3Packages
in the above does give you spliced packages.This happens because of the way splicing operates on derivations: https://github.com/NixOS/nixpkgs/blob/6796675202b8d32e151a31ec71218cff04b2447a/pkgs/top-level/splice.nix#L47-L82
Only the output attributes of a derivation are spliced; all other attributes are passed through as is.
Attrsets, on the other hand, are spliced recursively (on all attributes): https://github.com/NixOS/nixpkgs/blob/6796675202b8d32e151a31ec71218cff04b2447a/pkgs/top-level/splice.nix#L83-L98
Tying this back to the example above,
python3Packages
(despite being defined as an alias topython3.pkgs
) is spliced because the splice function is passed that attrset when recursively processing the top-level package attrset (whereas when the slice function encounterspython3
it does not recurse intopython3.pkgs
becausepython3
is a derivation).Potential Solutions
1: Have
pkgs
be already-spliced in such placesi.e.: instead of relying on the top-level splicing to splice such package sets, have this be the responsibility of the package set's scope.
Tangent: Splicing and Scopes
There's already some precedent for this; today scopes that use
makeScopeWithSplicing
are automatically given anewScope
and acallPackage
that contains spliced versions of the packages in the scope: https://github.com/NixOS/nixpkgs/blob/6796675202b8d32e151a31ec71218cff04b2447a/lib/customisation.nix#L280-L305Here's how the python scope is set up, for example:
pkgsBuildBuild
,pkgsBuildHost
, etc. sets are given tomakeScopeWithSplicing
; this yields thepythonPackage
fixed point: https://github.com/NixOS/nixpkgs/blob/7f6ecd4237de0369562dd120fb774467d0845819/pkgs/development/interpreters/python/default.nix#L43-L77python3
derivation aspkgs
: https://github.com/NixOS/nixpkgs/blob/7f6ecd4237de0369562dd120fb774467d0845819/pkgs/development/interpreters/python/default.nix#L97Crucially, the spliced packages
makeScopeWithSplicing
produces are not made available in the actual scope (i.e. the package set;python3.pkgs
in the above) itself but are made available to packages within the scope via thecallPackage
machinery.This actually means that the contents of scopes like
python3Packages
(i.e. scopes that are exposed directly – not via a derivation – and are spliced at the top-level) are actually spliced "twice": once as part of themakeScopeWithSplicing
call (accessed by members of the scope) and then again at the top-level (accessed by members of the outermost scope, the top-level).It would be nice if we could reuse the splicing.
Using the outer scope's splicing within the inner scope is easy to do:
__splicedPackages
to this function__splicedPackages. ...
in lieu ofspliced0
But this is problematic in cases where the scope is overriden (i.e.
overrideScope
) outside of an overlay-like context (where the top-level binding of that scope is also updated). In such cases, the scope'scallPackage
will continue using the original pre-overrideScope
version of the packages within the scope because the splicing would still be pulling things from the top-level__splicedPackages
. Put another way:overlays = [(f: p: { python3Packages = p.python3Packages.overrideScope (_: _: { ... }); })]
would be okay becausenixpkgs.python3Packages
is updated to point to the overriden scope but just doing(nixpkgs.python3Packages.overrideScope (_: _: { ... })).some-package
would not be okay.Going the other way (using the scope's splicing for the top-level) seems a little trickier but wouldn't have this issue. We'd need to have the
makeScopeWithSplicing
expose the spliced attrset with something like a__splicedPackages
attr and we'd then wantspliceReal
's handling of attrset to check for such an attr and use it instead of redoing the splicing itself. I don't think this runs into any recursion issues but I have not tested this yet.Ultimately this (splicing scopes multiple times) is somewhat orthogonal to this issue but the above has some overlap with the potential solutions below and might influence a decision there.
Now that we know how scopes get spliced:
The most straight-forward way to have our scope yield an already-spliced
pkgs
attr is probably to getmakeScopeWithSpliced
to give us it's spliced attrset. We can modifymakeScopeWithSpliced
to expose the attrset as__splicedPackages = spliced
(as discussed above) and then swap out this line: https://github.com/NixOS/nixpkgs/blob/7f6ecd4237de0369562dd120fb774467d0845819/pkgs/development/interpreters/python/default.nix#L97for
pkgs = pythonPackages.__splicedPackages
.We would also have to replicate this change for all other users of
makeScopeWithSpliced
that export their package set as part of a derivation's attrs.2: Adjust
spliceReal
's handling of derivationsRecursing on all of the attributes of every derivation seems fraught but maybe it's safe to recurse on
drv.passthru
or an opt-in list of attributes (i.e. we could havespliceReal
look for a passthru attr named__spliceRecurseAttrs
on derivations) or maybe even justpkgs
(since that seems to be the convention used).This has the benefit of not requiring any changes from users of
makeScopeWithSpliced
and handling splicing for packages that are referenced via a derivation's attrs (depending on how general we adjustspliceReal
to be on derivation attrs).3: Discourage using package sets like
python3.pkgs
"directly"(and instead push people to use
python3Packages
,lua5Packages
, etc. in nixpkgs)This seems suboptimal, both because this will be another thing that'd need to be enforced in nixpkgs to have cross work for packages and because the
foo.pkgs.bar
pattern (wherefoo
is a derivation) seems pretty pervasive in nixpkgs (python
,lua
,perl
,postgresql
, etc.).A version of option 2 (with the
__splicedPackages
attr for deduplicating the work of splicing scopes as a follow-up PR if it doesn't cause breakage) seems like the least-worst fix to me but I'm not particularly satisfied with any of these solutions; hopefully there's a more elegant solution that I'm missing 🤞.cc: @Artturin