search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.31k stars 13.54k forks source link

Package sets within derivations (i.e. `python3.pkgs`) are not spliced #211340

Open rrbutani opened 1 year ago

rrbutani commented 1 year ago

I ran into this while trying to get LLVM to eval under cross-compilation as part of #194634.

Apologies if this has already been addressed elsewhere; I know @Artturin has been working on fixes and cleanup for splicing recently but I didn't see this specific issue brought up in any of the existing issues and PRs.

Problem

As detailed in https://github.com/NixOS/nixpkgs/pull/194634/commits/888f966869d66b85042dc0971d1fc046578bf6f9, packages that are in package sets within derivations (i.e. the psutil package within python3.pkgs where python3 is a derivation) do not get spliced.

This can be observed via __splicedPackages (I know this is meant to be an implementation detail but it is the attrset that top-level callPackage draws from and it's handy for debugging):

> np = (import <nixpkgs> { }).pkgsCross.aarch64-multiplatform

> np.__splicedPackages.hello ? __spliced
true

> np.__splicedPackages.python3Packages.psutil ? __spliced
true (!!)

> np.__splicedPackages.python3.pkgs.psutil ? __spliced
false

Note that accessing the python package set via python3Packages in the above does give you spliced packages.

This happens because of the way splicing operates on derivations: https://github.com/NixOS/nixpkgs/blob/6796675202b8d32e151a31ec71218cff04b2447a/pkgs/top-level/splice.nix#L47-L82

Only the output attributes of a derivation are spliced; all other attributes are passed through as is.

Attrsets, on the other hand, are spliced recursively (on all attributes): https://github.com/NixOS/nixpkgs/blob/6796675202b8d32e151a31ec71218cff04b2447a/pkgs/top-level/splice.nix#L83-L98

Tying this back to the example above, python3Packages (despite being defined as an alias to python3.pkgs) is spliced because the splice function is passed that attrset when recursively processing the top-level package attrset (whereas when the slice function encounters python3 it does not recurse into python3.pkgs because python3 is a derivation).

Potential Solutions

1: Have pkgs be already-spliced in such places

i.e.: instead of relying on the top-level splicing to splice such package sets, have this be the responsibility of the package set's scope.

Tangent: Splicing and Scopes

There's already some precedent for this; today scopes that use makeScopeWithSplicing are automatically given a newScope and a callPackage that contains spliced versions of the packages in the scope: https://github.com/NixOS/nixpkgs/blob/6796675202b8d32e151a31ec71218cff04b2447a/lib/customisation.nix#L280-L305

Here's how the python scope is set up, for example:

Crucially, the spliced packages makeScopeWithSplicing produces are not made available in the actual scope (i.e. the package set; python3.pkgs in the above) itself but are made available to packages within the scope via the callPackage machinery.

This actually means that the contents of scopes like python3Packages (i.e. scopes that are exposed directly – not via a derivation – and are spliced at the top-level) are actually spliced "twice": once as part of the makeScopeWithSplicing call (accessed by members of the scope) and then again at the top-level (accessed by members of the outermost scope, the top-level).

It would be nice if we could reuse the splicing.

Using the outer scope's splicing within the inner scope is easy to do:

But this is problematic in cases where the scope is overriden (i.e. overrideScope) outside of an overlay-like context (where the top-level binding of that scope is also updated). In such cases, the scope's callPackage will continue using the original pre-overrideScope version of the packages within the scope because the splicing would still be pulling things from the top-level __splicedPackages. Put another way: overlays = [(f: p: { python3Packages = p.python3Packages.overrideScope (_: _: { ... }); })] would be okay because nixpkgs.python3Packages is updated to point to the overriden scope but just doing (nixpkgs.python3Packages.overrideScope (_: _: { ... })).some-package would not be okay.

Going the other way (using the scope's splicing for the top-level) seems a little trickier but wouldn't have this issue. We'd need to have the makeScopeWithSplicing expose the spliced attrset with something like a __splicedPackages attr and we'd then want spliceReal's handling of attrset to check for such an attr and use it instead of redoing the splicing itself. I don't think this runs into any recursion issues but I have not tested this yet.

Ultimately this (splicing scopes multiple times) is somewhat orthogonal to this issue but the above has some overlap with the potential solutions below and might influence a decision there.

Now that we know how scopes get spliced:

The most straight-forward way to have our scope yield an already-spliced pkgs attr is probably to get makeScopeWithSpliced to give us it's spliced attrset. We can modify makeScopeWithSpliced to expose the attrset as __splicedPackages = spliced (as discussed above) and then swap out this line: https://github.com/NixOS/nixpkgs/blob/7f6ecd4237de0369562dd120fb774467d0845819/pkgs/development/interpreters/python/default.nix#L97

for pkgs = pythonPackages.__splicedPackages.

We would also have to replicate this change for all other users of makeScopeWithSpliced that export their package set as part of a derivation's attrs.

2: Adjust spliceReal's handling of derivations

Recursing on all of the attributes of every derivation seems fraught but maybe it's safe to recurse on drv.passthru or an opt-in list of attributes (i.e. we could have spliceReal look for a passthru attr named __spliceRecurseAttrs on derivations) or maybe even just pkgs (since that seems to be the convention used).

This has the benefit of not requiring any changes from users of makeScopeWithSpliced and handling splicing for packages that are referenced via a derivation's attrs (depending on how general we adjust spliceReal to be on derivation attrs).

3: Discourage using package sets like python3.pkgs "directly"

(and instead push people to use python3Packages, lua5Packages, etc. in nixpkgs)

This seems suboptimal, both because this will be another thing that'd need to be enforced in nixpkgs to have cross work for packages and because the foo.pkgs.bar pattern (where foo is a derivation) seems pretty pervasive in nixpkgs (python, lua, perl, postgresql, etc.).

A version of option 2 (with the __splicedPackages attr for deduplicating the work of splicing scopes as a follow-up PR if it doesn't cause breakage) seems like the least-worst fix to me but I'm not particularly satisfied with any of these solutions; hopefully there's a more elegant solution that I'm missing 🤞.

cc: @Artturin

ghost commented 1 year ago

Potential Solutions

4. Reconsider whether or not splicing was a good idea

There has got to be a simpler way of doing all of this. The fact that splicing breaks randomly in weird ways and hardly anybody understands it is a major contributor to people hating on cross compilation. There has got to be a simpler way to do all of this.

Artturin commented 1 year ago

Potential Solutions

4. Reconsider whether or not splicing was a good idea

There has got to be a simpler way of doing all of this. The fact that splicing breaks randomly in weird ways and hardly anybody understands it is a major contributor to people hating on cross compilation. There has got to be a simpler way to do all of this.

https://github.com/NixOS/nixpkgs/issues/204303#issuecomment-1403662186

FRidh commented 1 year ago

3: Discourage using package sets like python3.pkgs "directly"

I think about 8 years ago I introduced this and it has been a clear mistake. Since then this pattern was copied throughout. We should indeed go to pythonPackages and also move the helpers such as buildEnv and withPackages into the package set, like haskell already had before.

AndersonTorres commented 1 year ago

@FRidh what are the costs of reverting it?

FRidh commented 1 year ago

@FRidh what are the costs of reverting it?

Basically people willing to put in that effort.

AndersonTorres commented 1 year ago

"Willing" also includes having some theoretical knowledge?

nixos-discourse commented 1 month ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/frustrations-about-splicing/49607/1