machine configs ignore overlay

[nh2]

I've defined a nixpkgs overlay but nixops isn't picking it up.

I'm using the method where I set NIX_PATH / -I to MYDIR where MYDIR contains MYDIR/nixpkgs-overlays/default.nix with contents like these. Because nix searches in $NIX_PATH/nixpkgs-overlays/default.nix by default, that overlay is picked up.

So when I pkgs = import <nixpkgs> {};, that pkgs contains my overrides.

But the pkgs passed to machine configs like machine1 = { pkgs, config, ... }: ... do NOT get passed my overrides, they get passed an un-overridden nixpkgs.

Why is that?

I can work around it by setting

machine1 = { pkgs, config, ... }: {
  nixpkgs.config.packageOverrides = pkgs: (import <nixpkgs> {});
  ...
}

but it seems overlays are supposed to work without doing this manually for every machine.

[nh2]

It seems I can fix it by changing this https://github.com/NixOS/nixops/blob/b7d549143451c9cfae6644db89a495de67df1407/nix/eval-machine-info.nix#L35-L66

to have

      { name = machineName;
        value = import <nixpkgs/nixos/lib/eval-config.nix> {
          pkgs = import <nixpkgs> {};          # new line addded here
          modules =
            modules ++
            ...

But not sure if that makes any issues. The relevant code for eval-config.nix doesn't even seem to know if this pkgs argument should be removed:

, # !!! is this argument needed any more? The pkgs argument can
  # be set modularly anyway.
pkgs ? null

[nh2]

@cleverca22 Explained this to me, will write it up tomorrow.

[nh2]

@nh2

are you using overlays? They seem to not work for me for nixops unless I patch this: https://github.com/NixOS/nixops/issues/893

---

@cleverca22

i believe that is a nixos issue, not a nixops issue you must set nixpkgs.overlays = [ (import ./overlay1.nix) ]; in the nixos config

---

@nh2

I must set that somewhere or that should be set somewhere in nixpkgs?

---

@cleverca22

you have to set it within the config for a machine in nixops

{
  machine1 = { ... }: {
    deployment.targetEnv = "ec2";
    nixpkgs.overlays = [ (import ./overlay1.nix) ];
  };
}

---

@nh2

Hmm. I had hoped that using overlays I could add overrides for my entire nixpkgs, so that is not the case and there's a separation between "nixops-evaluation-host-side pkgs" and "machine-host-side pkgs" in regards to overlays?

---

@cleverca22

there is also a special default machine in nixops that applies to everything

{
  default = { ... }: {
    nixpkgs.overlays = [ (import ./overlay1.nix) ];
  };
}

now every machine in the deployment will have that overlay

---

@nh2

you mean defaults with s? (at least that's what I'm using)

---

@cleverca22

ah, yeah, it says defaults in my netbook deployment

---

@nh2

hmm OK. But why is it like this, is this desired? I'm thinking of a use case, why it shouldn't automatically take my overlays

purity, nixos should use the config/overlays defined in the nixos config, not whatever is in $HOME of the current user the code responsible for loading the pkgs arg in nixos is at https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/nixpkgs.nix anything assigned to _module.args is passed to every module in the OS eval line 64 is where the default nixpkgs comes from ah, the pkgs argument you found in eval-config overrides that, causing both nixpkgs.config and nixpkgs.overlays to do nothing but setting it under defaults.nixpkgs.overlays allows it to compose against machine1.nixpkgs.overlays, so you still have more flexible changes

---

@nh2

I still find the argumentation slightly confusing though: When it comes to the evaluation of import <nixpkgs>, that nixpkgs is taken from NIX_PATH. In https://nixos.org/nixpkgs/manual/#sec-overlays-install it says

If the overlays argument is not provided explicitly, we look for overlays in a path. The path is determined as follows: ... Otherwise, if the Nix path entry <nixpkgs-overlays> exists, we look for overlays at that path

So in my case it picks <nixpkgs> from NIX_PATH but not <nixpkgs-overlays> -- that seems to be against the above description.

---

@cleverca22

nixops may also be messing with NIX_PATH during the eval, try inserting builtins.trace (builtins.toJSON builtins.nixPath) "foo" into the codepath somewhere so you can see what the nixPath is

---

@nh2

OK one sec

---

@cleverca22

builtins.nixPath will contain the result of merging NIX_PATH with all the -I flags and the "foo" can be removed, since trace will return whatever is given to it, like in haskell and we have no IO monad, so it has to be something nixops will want to read

---

@nh2

it seems to not mess with it in this case, builtins.nixPath is the same both on the top-level as well as inside the machine config, namely in my case

[{"path":"/nix/store/z7g5gqdl9lv2zj35ylqg0jnhf29ijsxs-nixops-1.5.2pre0_abcdef/share/nix/nixops","prefix":"nixops"},{"path":".","prefix":""},{"path":"../../nix-channel","prefix":""},{"path":"/nix/store/b4s1gxiis1ryvybnjhdjvgc5sr1nq0ys-nix-1.11.15/share/nix/corepkgs","prefix":"nix"}]

---

@cleverca22

ok, so it will search for <nixpkgs-overlays> in the current directory and ../../nix-channel/nixpkgs-overlays

---

@nh2

yes that is right

---

@cleverca22

and does it exist in the current directory?

---

@nh2

no, only in ../../nix-channel

---

@cleverca22

ah, you have a channel called nixpkgs-overlays? ls -lh ../../nix-channel/{,nixpkgs-overlays}

---

@nh2

I just called the dir that I put in NIX_PATH "nix-channel"

---

@cleverca22

ah what does the above ls say?

---

@nh2

% ls -lah nix-channel
total 16K
drwxrwxr-x  4 niklas niklas 4.0K Mar  4 01:03 ./
drwxr-xr-x 37 niklas niklas 4.0K Mar  3 23:56 ../
drwxrwxr-x  8 niklas niklas 4.0K Mar  3 15:14 nixpkgs/
drwxrwxr-x  2 niklas niklas 4.0K Mar  4 01:03 nixpkgs-overlays/

% ls nix-channel/nixpkgs-overlays
default.nix

---

@cleverca22

https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/impure.nix#L53 
ok, so all files matching `*.nix` and all directories matching `*/default.nix` will be passed to import, to build up the overlay list
that should work

---

@nh2

@cleverca what would happen if we changed nixops as I proposed in that issue, and then somebody set nixpkgs.overlays = [ ]; inside a machine config? Would that remove the overlays loaded from NIX_PATH I added with my change? Because if yes, maybe then what I propose there is a better default: They would be there by default as described in the nixpkgs manual, and if somebody wanted to NOT have those auto-added-from-NIX_PATH overlays on one of their machines, they could set nixpkgs.overlays = []

---

@cleverca22

nixpkgs would entirely ignore all config the users set at nixpkgs.overlays and nixpkgs.config causing confusing problems and the result would depend on the environment, rather then what is defined in the nix files

---

@nh2

but right now doesn't the result also depend on the environment, given that it picks <nixpkgs> from NIX_PATH?

---

@cleverca22

thats a much more expected thing, that can be overriden with nixops modify deployment.nix -I nixpkgs=https://....

---

@nh2

nixpkgs would entirely ignore all config the users set at nixpkgs.overlays

hmm you're right, I just tried it and if I do what I said, then setting nixpkgs.overlays = [ ]; has no effect

---

@cleverca22

you could even nixpkgs.overlays = throw "foo"; i think since its ignored to that level

---

@nh2

no that throws though

---

@cleverca22

ah, but that stack trace may say why

---

@nh2

maybe it does evaluate it but concatenate the results, so it's what I added ++ []? @nh2 (uploaded uploads file) and commented: that's when I do nixpkgs.overlays = throw "foo"; while having pkgs = import <nixpkgs> {}; set in eval-machine-info.nix

---

@cleverca22

ahh, i think i see why

---

@nh2

yeah I think it's because of https://github.com/NixOS/nixpkgs/blob/1dcd022f01b251b1656f349dcf749c0890de2799/nixos/modules/misc/nixpkgs.nix#L78

---

@nh2

type = types.listOf overlayType; so default concatenates

---

@cleverca22

its evaling both versions of pkgs, from the normal place, and your override, then its passing both sets of pkgs to the mkMerge system

---

@nh2

right

---

@cleverca22

but it wont concat the overlays its the pkgs that are being merged and ... https://github.com/NixOS/nixpkgs/blob/4eb7945515b260efa55e44e783f4b4936de9f6a4/nixos/lib/eval-config.nix#L40 if you supply a pkgs the way you said, it will be an mkForce so after it evals both, it throws out the one that obeys nixpkgs.overlays if you modify a package within it with a second overlay, you should find that it doesnt take effect

---

@nh2

wait, I don't follow. Why wouldn't it concat the overlays? Given that overlays = mkOption and type = types.listOf overlayType;, shouldn't the options merging code think "it's a list, the default merge operations for lists is concatenation"?

---

@cleverca22

it doesnt merge <nixpkgs-overlays> and nixpkgs.overlays it creates 2 entirely seperate instances of nixpkgs, one from each misc/nixpkgs.nix line 64 will import the 1st nixpkgs, which obeys nixpkgs.overlays and nixpkgs.config, and its at the default priority, because its in the default key of mkOption

---

@nh2

ah hmm

---

@cleverca22

lib/eval-config.nix line 40, then sets the same _module.args.pkgs option with mkForce so the 1st nixpkgs is entirely thrown out

---

@nh2

and if we set nixpkgs.overlays = [ ... some overlay import logic as described in the nixpkgs manual ... ] among these lines https://github.com/NixOS/nixops/blob/b7d549143451c9cfae6644db89a495de67df1407/nix/eval-machine-info.nix#L54-L60 would that work and make sense? so that it's an actual option that would get merged with concat, and could be overridden by the user for a machine with mkForce?

---

@cleverca22

there is no way to remove those, and mkForce removes all other versions, so i cant concat between all of my own files so i either have to live with whatever you added, or put all of my own overlays in a single file

---

@nh2

hmm OK, that doesn't sound good

---

@cleverca22

for example, i started this nixos module: https://github.com/cleverca22/nixos-configs/blob/e534f8706fae4f6178194418dab2cf1944a972eb/qemu.nix that will add its own overlay, which pulls in some extra packages it needs and i plan to have several modules like that, each with their own overlay

---

@nh2

then I think best is to add a section to the nixops manual that explains this, that only nixpkgs is taken from NIX_PATH for the machines, and that you should define nixpkgs.overlays accordingly per machine if you want the overlays to apply

---

@cleverca22

the same is true for nixos itself, so maybe add it to the nixos manual, and have a note in nixops pointing to that?

---

@nh2

Yes sounds good. The section in the nixpkgs manual hits a bit at it with> On a NixOS system the value of the nixpkgs.overlays option, if present, is passed to the system Nixpkgs directly as an argument. Note that this does not affect the overlays for non-NixOS operations (e.g. nix-env), which are looked up independently. but I think it could be clearer.

[nh2]

So the summary and remainder for this issue is to improve docs in nixops and nixpkgs accordingly.

Namely, you have to give nixpkgs.overlays = [ ... ] explicitly for your nixops machine (or do it in defaults if you want it for all machines in a network).

[rbrewer123]

I'm still new to NixOS and nixops, and I'm trying to run a customized version of zfs with nixops. I found this bug report, but I'm still getting an error about

nixops deploy -d zfs
building all machine configurations...
error: attribute 'extend' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/boot/kernel.nix:39:31
(use '--show-trace' to show detailed location information)
error: unable to build all machine configurations

when I deploy.

Here's the "logical" part of my nixops config:

{
  network.description = "zfs test";

  zfstest =
    { config, pkgs, ... }:
    { 
       boot.zfs.enableUnstable = true;
       boot.supportedFilesystems = [ "zfs" ];
       boot.kernelPackages = pkgs.linuxPackages_4_15;

        nixpkgs.overlays = [(
          self: super:
          {
            zfsUnstable = super.zfsUnstable.overrideAttrs (oldAttrs : {

              src = super.fetchFromGitHub {

                owner = "rbrewer123";
                repo = "zfs";

                #rev = "b4555c777a0be3c0dba29662d278c57099c60a87";
                #sha256 = "03k1maiz70jbk45619yqcf3larhvgilaxnvkid3972knm7ycgjsh";
                rev = "de4a0c31b5d269e39114181465ddbba448a4db3f";
                sha256 = "0ghpn9lsjdpx2qh4wk48vdd22sa1qv4hw49m0p4lv6ap5h14bx0v";
              };

            });

            linuxPackages_4_15.zfsUnstable = super.linuxPackages_4_15.zfsUnstable.overrideAttrs (oldAttrs : {

              src = super.fetchFromGitHub {

                owner = "rbrewer123";
                repo = "zfs";

                #rev = "b4555c777a0be3c0dba29662d278c57099c60a87";
                #sha256 = "03k1maiz70jbk45619yqcf3larhvgilaxnvkid3972knm7ycgjsh";
                rev = "de4a0c31b5d269e39114181465ddbba448a4db3f";
                sha256 = "0ghpn9lsjdpx2qh4wk48vdd22sa1qv4hw49m0p4lv6ap5h14bx0v";
              };

            });

          }
          )];
    };
}

Using those overlays alone, I'm able to build the userspace and kernel space portions of zfs with nix build. But when I put them in nixops as shown it doesn't work. Do you have any ideas what's wrong?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/can-we-add-integrations-to-discourse/156/1