Discourage/prohibit inheriting from package sets in callPackage

[Artturin]

in all-packages.nix when something is inherited (ex, {inherit (pkgs.darwin.apple_sdk.frameworks) Security;}) in a callPackage invocation the inherited attr will not have __spliced and so won't be cross-compilation compatible.

one way to fix this would be to change the with pkgs; in all-packages to with pkgs.__splicedPackages like tried in #58327 however that will result in other issues like infinite recursion.

$ nix build ".#pkgsCross.aarch64-multiplatform.bash"
error: infinite recursion encountered

other way to work around it would be to to inherit from the __splicedPackages set (ex, {inherit (pkgs.__splicedPackages.darwin.apple_sdk.frameworks) Security;})

the way i prefer would be to just not inherit from package sets

Demo: show-splices.nix

let
  pkgs = import ./. { };
  pkgsCross = pkgs.pkgsCross.aarch64-multiplatform;
  workingSplices = pkgsCross.callPackage
    ({ darwin, xorg }: {
      d = darwin.apple_sdk.frameworks.Security.__spliced;
      x = xorg.libX11.__spliced;
    })
    { };

  notWorkingSplices = pkgsCross.callPackage
    ({ Security, libX11 }: {
      d = Security.__spliced;
      x = libX11.__spliced;
    })
    {
      inherit (pkgsCross.darwin.apple_sdk.frameworks) Security;
      inherit (pkgsCross.xorg) libX11;
    };
in
{
  # for --json to work
  workingSplices =
    removeAttrs workingSplices [ "override" "overrideDerivation" ];
  notWorkingSplices =
    removeAttrs notWorkingSplices [ "override" "overrideDerivation" ];
}

working:

$ NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix eval -f show-splices.nix workingSplices --json | jq
{
  "d": {
    "buildBuild": "/nix/store/ls83bixygn2bzqq1w2z66dyccfn61wic-apple-framework-Security",
    "buildHost": "/nix/store/ls83bixygn2bzqq1w2z66dyccfn61wic-apple-framework-Security",
    "buildTarget": "/nix/store/ls83bixygn2bzqq1w2z66dyccfn61wic-apple-framework-Security",
    "hostHost": "/nix/store/vhqjns1zfab6cnv2lbkr6gff1fa8n3kw-apple-framework-Security-11.0.0",
    "hostTarget": "/nix/store/vhqjns1zfab6cnv2lbkr6gff1fa8n3kw-apple-framework-Security-11.0.0"
  },
  "x": {
    "buildBuild": "/nix/store/w3zzhfl4a7xp0xfflz2gawv02y8ba9z8-libX11-1.8.1",
    "buildHost": "/nix/store/w3zzhfl4a7xp0xfflz2gawv02y8ba9z8-libX11-1.8.1",
    "buildTarget": "/nix/store/w3zzhfl4a7xp0xfflz2gawv02y8ba9z8-libX11-1.8.1",
    "hostHost": "/nix/store/gjh7n6myi7miwprjkk4lll8cp1j7kmz8-libX11-aarch64-unknown-linux-gnu-1.8.1",
    "hostTarget": "/nix/store/gjh7n6myi7miwprjkk4lll8cp1j7kmz8-libX11-aarch64-unknown-linux-gnu-1.8.1"
  }
}

broken:

}
$ NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix eval -f show-splices.nix notWorkingSplices --json | jq
error: attribute '__spliced' missing

       at /home/artturin/nixgits/my-nixpkgs/show-splices.nix:5:74:

            4|   workingSplices = pkgsCross.callPackage ({ darwin, xorg }: { d = darwin.apple_sdk.frameworks.Security.__spliced; x = xorg.libX11.__spliced;}) { };
            5|   notWorkingSplices = pkgsCross.callPackage ({ Security, libX11 }: { d = Security.__spliced; x = libX11.__spliced;}) { inherit (pkgsCross.darwin.apple_sdk.frameworks) Security; inherit (pkgsCross.xorg) libX11; };
             |                                                                          ^
            6| in
{
  "d": null
}

related issues: https://github.com/NixOS/nixpkgs/issues/49526

[AndersonTorres]

@Artturin , please, can you (or someone) explain what is the matter around the splicing?

I have seen today some code on XFCE mega-attrset, and I am confused.

Also, I have some packages that use Darwin libs, so I will start to cleanup them.

[Artturin]

@Artturin , please, can you (or someone) explain what is the matter around the splicing?

splicing adds the __spliced attribute set to the package mkDerivation then picks __spliced.buildHost for nativeBuildInputs https://github.com/NixOS/nixpkgs/blob/86c83ab79d71084b4b107adf4c3838b7cbcc6446/pkgs/stdenv/generic/make-derivation.nix#L212

and __spliced.hostTarget for buildInputs https://github.com/NixOS/nixpkgs/blob/86c83ab79d71084b4b107adf4c3838b7cbcc6446/pkgs/stdenv/generic/make-derivation.nix#L212

[sternenseemann]

I think it is not possible to get rid of it entirely, especially if you consider the API benefit of being able to override individual inputs which is certainly an intention in some of the cases. Also, in sub package sets the solution you propose in #195874 won't work, of course.

[tjni]

There's some magic going on with __spliced that's not obvious to me. Given that, it sounds hard to teach to contributors, so how do we prohibit this pattern?

[Atemu]

I concur that, ideally, we do not want to pass sets of packages to packages. Same reason we don't just pass pkgs everywhere and call it a day.

Why are the Darwin frameworks hidden away in a subset anyways? If there's no good reason for it, we might want to consider just adding them to the top-level.

Another thing we could consider is a special attribute where we can simply pass attribute sets from which attrs should be considered as if they were at the top-level but at that point, we might as well use the top-level in the first place?

[AndersonTorres]

Why are the Darwin frameworks hidden away in a subset anyways?

I don't believe they are hidden, properly speaking. Anyone can access them by the cumbersome inherit mechanism. The problem to me is the magic behind them and that huge amount of if isDarwin then [ Security OpenGL Metal ... ]

If there's no good reason for it, we might want to consider just adding them to the top-level.

This is a bit personal and preciousness-y, but I do not like the currently huge top-level file and its propensity to scramble the asciibetical ordering.

[Artturin]

stuff like this should be discouraged/prohibited too

  keybinder = callPackage ../development/libraries/keybinder {
    automake = automake111x;
    lua = lua5_1;
  };

nix-repl> pkgsCross.aarch64-multiplatform.keybinder.nativeBuildInputs
[ ... «derivation /nix/store/yj8vj3gbd78scrj4xjmdx4rm5p87dyb9-automake-aarch64-unknown-linux-gnu-1.11.6.drv» ]

if it has to be done then it should be

  keybinder = callPackage ../development/libraries/keybinder {
    automake = __splicedPackages.automake111x;
    lua = __splicedPackages.lua5_1;
  };

nix-repl> pkgsCross.aarch64-multiplatform.keybinder.nativeBuildInputs
[ ... «derivation /nix/store/c9g2kiyrwv9hq33wa3ckrkz95y4dq1qh-automake-1.11.6.drv» ]

[figsoda]

can we fix that by changing (all-packages.nix L13)

with pkgs;

to

with pkgs.__splicedPackages;

?

[Artturin]

i linked https://github.com/NixOS/nixpkgs/pull/58327 in the OP

a thing we could try is reorganize all-packages.nix so we don't get infinite recursion

quick POC (added with pkgs.__splicedPackages under wrapBintoolsWith which was giving inf rec)

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 5e0e798be20..765ce0b92c0 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -8,7 +8,7 @@
 { lib, noSysDirs, config, overlays }:
 res: pkgs: super:

-with pkgs;
+(with pkgs;

 {
   # A module system style type tag
@@ -15618,6 +15618,9 @@ with pkgs;

   yaml-language-server = nodePackages.yaml-language-server;

+}) // (with pkgs.__splicedPackages; {
+
+
   # prolog
   yap = callPackage ../development/compilers/yap { };

@@ -38170,4 +38173,4 @@ with pkgs;
   aitrack = libsForQt5.callPackage ../applications/misc/aitrack { };

   widevine-cdm = callPackage ../applications/networking/browsers/misc/widevine-cdm.nix { };
-}
+})

a issue with that is the use of // so we won't get errors about duplicate attrs and the RHS will override the LHS

[AndersonTorres]

https://github.com/NixOS/nixpkgs/pull/204370

[Atemu]

I don't believe they are hidden, properly speaking. Anyone can access them by the cumbersome inherit mechanism.

That's what I mean.

I do not like the currently huge top-level file and its propensity to scramble the asciibetical ordering.

Note that I said "top-level", not "top-level file". I don't care what file they're in but I do want them to be accessible from pkgs so that the auto-discovery mechanism, splicing and all the other quirks of callPackage's "just work" with them.

Alternatively, an extra attr that automagically extends the main package set for the purpose of discovering dependencies for that specific package would also be fine in my eyes. i.e.

  bar = callPackage ../foo/bar {
    extraPackageSets = [ darwin xorg ];
  }

{
  stdenv,
  libX11,
  Security,
  ...
}

...

stuff like this should be discouraged/prohibited too

  keybinder = callPackage ../development/libraries/keybinder {
    automake = automake111x;
    lua = lua5_1;
  };

In cases like this, I prefer to simply reference automake111x and lua5_1 explicitly in the inputs of the package's function. I.e.

{ 
  lib,
  stdenv,
  automake111x,
  lua5_1,
  ...
}:

let
  lua = lua5_1;
  automake = automake111x;
in

stdenv.mkDerivation {
  nativeBuildInputs = [ automake lua ];
}

(Or just in-line without the let.)

IMO, we should have as little logic as possible in all-packages.

[AndersonTorres]

It's becoming confusing...

It should be something more, ehr, ergonomical. Yes, I believe we should be more cross-compilaton-aware and we should strive to make it a "first class citizen" in Nix{,pkgs}. However, should this be engulfed by magical syntax/semantics?

Or maybe there is a terminology we all should learn?

[Atemu]

The most ergonomic will always be to not have package sets and have everything under the top-level as I initially suggested.

My alternative suggestion is just something that came to my mind because we might need something like that anyways because we do have huge package sets whose packages other packages depend on and, unlike the smaller ones we've been discussing so far, we probably don't want to dump those into the top-level.
If you can think of a more ergonomic solution to that problem, I'm all ears.

[AndersonTorres]

Taking the idea to a ridiculous extreme, all the expressions organized in subdirectories can be "concatenated" in a huge 4GB single file, sqlite-amalgam style. Indeed, I have refactored some Elisp packages that were written directly onto manual-packages.nix some time ago.

Thinking a bit about it, this makes few to no sense. We have custom code to shelter platforms like FreeBSD and PowerPC, it would be insane to make all of it accessible to the top-level.

Edit: I found the word: uniformization!

[viraptor]

Why are the Darwin frameworks hidden away in a subset anyways?

The frameworks come from different sdks, so sometimes we want the ones from apple_sdk, sometimes apple_sdk_11_0 (and there will be more). (On a second read... did you ask why the sdks are under Darwin, not top level?)

[AndersonTorres]

(On a second read... did you ask why the sdks are under Darwin, not top level?)

In a certain sense, yes.

[lilyinstarlight]

a issue with that is the use of // so we won't get errors about duplicate attrs and the RHS will override the LHS

@Artturin - I'm a bit late in responding to this, but don't we have lib.attrsets.unionOfDisjoint for this reason (which throws when the sets intersect) if we wanted to go the route of that POC you posted above?

[rnhmjoj]

To me this looks more like a deficiency in the way cross-compilation has been implemented (splicing). Is there really no alternative that would work with inheriting attributes? Because what you say should be discouraged/prohibited seems pretty useful.

[Artturin]

To me this looks more like a deficiency in the way cross-compilation has been implemented (splicing). Is there really no alternative that would work with inheriting attributes? Because what you say should be discouraged/prohibited seems pretty useful.

In all-packages where with pkgs is used something like https://github.com/NixOS/nixpkgs/issues/204303#issuecomment-1336269127 would fix it there

Everywhere else we just need to use callPackage like we already are (or should be)

[AndersonTorres]

a issue with that is the use of // so we won't get errors about duplicate attrs and the RHS will override the LHS

https://nix.dev/anti-patterns/language#attr1-attr2-merge-operator

[Atemu]

Another idea I've had here would be to consequently use callPackage wrappers. This is already being done to some degree by i.e. the QT packages but I see no reason why we couldn't expand this pattern to Darwin packages, xorg packages and everything else:

  bar = callPackage ../foo/bar {
    inherit (darwin.apple_sdk.frameworks) AppKit;
  };

->

  bar = darwin.apple_sdk.callPackage ../foo/bar { };

The example package here would reference AppKit in its args and darwin.callPackage would do callPackage but add the (properly spliced) darwin frameworks.

We could generalise this callPackage wrapper creation to allow for ad-hoc building of custom wrappers for more complicated packages which depend on multiple package sets:

  bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

and then the derivation could reference both Darwin frameworks and xorg packages in its args.

What do you think?

[AndersonTorres]

  bar = darwin.apple_sdk.callPackage ../foo/bar { };

I believe this is, strong words follow, madness. Why should I call a specific function hidden inside the Darwin subset in order to build non-Darwin stuff?

Yes, "this function will delegate to normal callPackage when building nonDarwin", but still it is not something well enginneered. This is hacky.

  bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

and then the derivation could reference both Darwin frameworks and xorg packages in its args.

What do you think?

Way better! +1 for this.

[figsoda]

Why should I call a specific function hidden inside the Darwin subset in order to build non-Darwin stuff?

We already do that for a lot of things that require apple sdk 11.0 but are not specific to darwin, search darwin.apple_sdk_11_0.callPackage

bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

How do we know what the attribute names of the packages are supposed to be, and how does it differ from the current callPackage exactly?

[Atemu]

Yes, "this function will delegate to normal callPackage when building nonDarwin", but still it is not something well enginneered. This is hacky.

Well, darwin.apple_sdk.callPackage would be defined as callPackageWith [ darwin.apple_sdk.frameworks ]. It'd just be a handy shorthand to not have to type that out every time because calling a package with darwin frameworks is a very common occurrence.

How do we know wait the attribute names of the packages are supposed to be

I'm not sure what you mean? The attrs given to the function are the sets of packages on whose attrs you can depend in the derivation.

You depend on the individual packages just like before.

and how does it differ from the current callPackage exactly?

It actually doesn't differ much; callPackage is defined in such a manner already. These functions I mentioned mostly already exists actually (under different names), we just need to wire them up.

[Artturin]

How do we know wait the attribute names of the packages are supposed to be

I'm not sure what you mean? The attrs given to the function are the sets of packages on whose attrs you can depend in the derivation.

The sets will have to be passed as strings to we can get them from __splicedPackages

[AndersonTorres]

We already do that for a lot of things that require apple sdk 11.0 but are not specific to darwin, search darwin.apple_sdk_11_0.callPackage

And this is the problem I am pointing out.

How do we know wait the attribute names of the packages are supposed to be, and how does it differ from the current callPackage exactly?

How do I know what callPackage to use, after all? The one exposed in the top-level? The one buried on Darwin toolchain - after two indirections?

Or, in the foreseeable future when Nix/Nixpkgs tools cover other OSes, the one buried on BSD, after God knows how much indirection?

[Artturin]

Yes, "this function will delegate to normal callPackage when building nonDarwin", but still it is not something well enginneered. This is hacky.

  bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

and then the derivation could reference both Darwin frameworks and xorg packages in its args. What do you think?

Way better! +1 for this.

POC https://github.com/NixOS/nixpkgs/pull/210970

[Ericson2314]

I made splicing (nativeDrv and crossDrv are older, but were done differently). Splicing is bad. I never wanted it. I causes major headaches when getting anywhere near bootstrapping. It is gross. It is probably slow. It should be removed.

[viraptor]

While I don't have a better solution to propose now, just wanted to mention that using both a different apple_sdk version and using Qt libs does not have an obvious example (you can list them explicitly in the set after callPackage, but I'm likely not the only person who wondered "how do I do the equivalent of qt5.callPackage and darwin.apple_sdk_11_0.callPackage at the same time". (without reading the implementation and understanding the internals) callPackageWith would make it more explicit. There are more combinations with other namespaces too, since we've got darwin.apple_sdk.frameworks, but also darwin.apple_sdk.libs

So, a more composable / approachable way would be great.

[lilyinstarlight]

It may be just because I don't know what I'm doing, but do we really need more than this to change the with pkgs; at the top of all-packages.nix to with pkgs.__splicedPackages;?

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index bb5f941b0a2..28786dc713f 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -8,7 +8,7 @@
 { lib, noSysDirs, config, overlays }:
 res: pkgs: super:

-with pkgs;
+with pkgs.__splicedPackages;

 {
   # A module system style type tag
@@ -14205,6 +14205,7 @@ with pkgs;
   # The GCC used to build libc for the target platform. Normal gccs will be
   # built with, and use, that cross-compiled libc.
   gccCrossStageStatic = assert stdenv.targetPlatform != stdenv.hostPlatform; let
+    binutilsNoLibc = pkgsHostTarget.binutilsNoLibc;
     libcCross1 = binutilsNoLibc.libc;
     in wrapCCWith {
       cc = gccFun {

A handful of tests for standard and cross attrs indicate it evaluates to the same store paths as before, including for those with the problematic dependency paths (like nix which uses pkgsStatic.busybox). Running nixpkgs-review also successfully evals the whole set and reports no changed packages

That said, I would still like to have callPackageWithExtraSet like @Artturin offered regardless of this

[leifhelm]

With callPackagesWithExtraSet how would pkgs.python310Packages.zstd specify that it does not need itself but pkgs.zstd?

[Atemu]

Something like

  callPackagesWithExtraSets [ pkgs python3Packages ] ...;

or

  callPackagesWithExtraSet (python3Packages // { inherit (pkgs) zstd; }) ...;

could work. Here, pkgs would take precedence over python.

[Artturin]

With callPackagesWithExtraSet how would pkgs.python310Packages.zstd specify that it does not need itself but pkgs.zstd?

the bracket part of callPackage ./blah { } would still function normally so you would just inherit from __splicedPackages

[fgaz]

I also think splicing should be removed, in favor of something more explicit or less leaky.

I have a rough idea of what I think should replace it (I may have mentioned this to @Ericson2314 already). I meant to write a longer proposal, but since we're having this discussion here's a summary:

• In the most common case, packages would take two sets of arguments: buildHost stuff and hostTarget stuff. More complex packages could take even more sets of arguments.

  { lib
  , stdenv
  , cmake
  , sometool
  }:
  { libfoo
  , libbar
  }:
  stdenv.mkDerivation ...

  • Some sort of deep pattern matching in the nix language would help with this. Instead of multiple (easy to confuse) arguments, we could have

    { lib # ← could be in a "platformIndependent" set instead of top level
    , buildHost =
    { stdenv
    , cmake
    , sometool
    }
    , hostTarget =
    { libfoo
    , libbar
    }
    }:
    stdenv.mkDerivation ...

• callPackage would pass buildHost packages as first argument and hostTarget packages as second.
  • There could be a callPackage variant with more platform combinations (the nix extension mentioned above would avoid this)
• buildInputs, nativeBuildInputs etc would be merged into a single inputs argument (EDIT by @Artturin They can't be merged because hook offsets need them.)
• There would be an override function for each input set: overrideBuildHost, overrideHostTarget...
• Existing packages could be migrated gradually either by giving the new callPackage a different name, like callPackageV2 or by making it detect when a package takes only a single attribute set.

[AndersonTorres]

It looks like meat for a future RFC!

[ghost]

I also think splicing should be removed, in favor of something more explicit or less leaky.

@fgaz I have been working on something similar. It's not finished, but I've posted the work-in-progress before I factor in your overrideHost/overrideBuild idea, which solves one of the problems I ran into. https://github.com/NixOS/nixpkgs/issues/227327

Thanks to @Artturin for letting me know about this.

[ghost]

when something is inherited (ex, {inherit (pkgs.darwin.apple_sdk.frameworks) Security}) in a callPackage invocation

Note that RFC 140 will deprecate callPackage _ { /* non-empty */ } so this problem might solve itself.

[infinisil]

@amjoseph-nixpkgs No it won't deprecate that, such invocations will still be valid! It's in scope to look at for the Nixpkgs Architecture Team, but until custom arguments have an adequate replacement of its use cases (which RFC 140 isn't), it shouldn't be deprecated

[AtnNn]

Some sort of deep pattern matching in the nix language would help with this

I've experimented with a similar approach that doesn't require changes to the Nix language. In https://github.com/NixOS/nixpkgs/compare/master...AtnNn:nixpkgs:deepcall callPackage looks for parameters that contain ' and "replaces" it with .. So packages can take as arguments llvmPackages'bintools, pkgsBuildHost'nodejs, perlPackages'CGIFast, targetPackages'stdenv'cc, xorg'libX11, python3Packages'docutils, gst_all_1'gstreamer, unixODBCDrivers'mariadb, etc.

[fgaz]

@AtnNn very nice, thanks for trying it out!

While yours is a clever solution given the current Nix language limitation, I think in the long run a feature like the one I proposed is necessary to keep everything clean. Nix is a domain specific language after all!

At the moment I'm busy, but I might draft an RFC later this year

[ghost]

until custom arguments have an adequate replacement of its use cases

What can callPackage _ { /* non-empty */ } do that (callPackage _ { }).override cannot?

Missing arguments are the only thing I can think of, and the easy workaround is to add ? throw "missing".

[AndersonTorres]

Thanks for remembering me about this! Now I need to rewrite something about getting rid of splicing.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-this-the-current-state-of-things/45944/14

[physics-enthusiast]

Random suggestion - how about implementing cross via an argument in mkDerivation? Have an offset parameter denoting the current derivation's position in the (build, ..., build, build, host, host, host, ..., host) chain which determines their internal platform triplet. Then dependent mkDerivations can call overrideAttrs on their nativeBuildInputs with an offset one less then their own, and on their depsTargetTarget with an offset one higher. Then, we no longer need to worry about callPackage injecting splices in all the places they're needed. As a bonus, external consumers of Nixpkgs expressions calling mkDerivation can participate in cross as dependencies without having to inject their own __spliced.buildHost and __spliced.hostTarget. I don't have full knowledge of the history of cross and why it is the way it is right now, so maybe there's some massive hidden downside I'm not aware of? More thunks, I guess?

[Atemu]

Then dependent mkDerivations can call overrideAttrs on their nativeBuildInputs with an offset one less then their own

I'm not sure how you're imagining this to work with overrideAttrs?

[physics-enthusiast]

I'm not sure how you're imagining this to work with overrideAttrs?

We'd add offset to derivationArg in make-derivation.nix, which I presume is how it becomes overrideable via overrideAttrs. To be a bit clearer, I'm proposing we add this lever to all mkDerivation calls everywhere; the universality is what makes it useful. Semantics would remain unchanged since we're still doing the exact same platform shuffling, only in a different place. Thinking about it a little bit more, we might also need a second offset to accomodate the more esoteric combinations like depsHostHost or depsBuildTarget.

Edit: Nevermind, I forgot about all of the variables defined outside the mkDerivation call. I think I see what you were getting at now.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/frustrations-about-splicing/49607/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/frustrations-about-splicing/49607/7

[reckenrode]

The way forward for Darwin is to get rid of frameworks and provide an SDK in the stdenv. It doesn’t help other package sets, but it should at least get rid of one of the major contributors to this problem.