Declarative containers ignore networking.nameservers, even with privateNetwork=true

delroth commented 2 years ago

Describe the bug

NixOS declarative containers have a "feature" that overwrites the container's /etc/resolv.conf at startup with the one from the host. I have doubts about that implementation in the general case, but in the case of a container configured with privateNetwork = true; it is beyond broken. Not only does it copy a resolv.conf that will in all likelihood not work (because we're in a private network namespace, with different interfaces / addressing), but it also somehow confuses resolvconf and makes networking.nameservers not do anything by default.

There is a workaround: environment.etc."resolv.conf".text = "..."; but that's very counter-intuitive to find, and we shouldn't require that for basic networking configuration.

I'm not entirely sure how to fix this, it seems like there are two issues compounded with each other here. First, I think copying resolv.conf with privateNetwork = true just shouldn't happen, if it ever does something useful I suspect that's accidental. Second, resolvconf should be overwriting that host resolv.conf file with what has been configured in networking.nameservers, but it doesn't do it, and I don't yet understand why. It just decides to not touch the file and copy it verbatim.

Steps To Reproduce

  containers.test = {
    privateNetwork = true;
    config = {
      networking.nameservers = [ "8.8.8.8" ];
    };

    # irrelevant to this bug
    autoStart = true;
    ephemeral = true;
  };

host# nixos-container root-login test
test# cat /etc/resolv.conf
# Generated by resolvconf
search delroth.net
nameserver 127.0.0.53
options edns0 trust-ad
options edns0

Expected behavior

A declarative container can define its own DNS configuration without getting trampled over by a bash script.

Screenshots

n/a

Additional context

The original implementation of the resolv.conf copying dates back to 2014: e620be97fe212b43ee42865dd4d3c8bba7f26fa9

Notify maintainers

Tagging @abbradar because I suspect from git history they might know about resolvconf, but that's a shot in the dark :-)

Metadata

system: "x86_64-linux"
host os: Linux 5.15.16-hardened1, NixOS, 22.05 (Quokka)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.7.0pre20220221_caf5172
channels(delroth): ""
nixpkgs: /etc/nixpkgs

aanderse commented 2 years ago

@Ma27 does your rewrite of the containers have this same behaviour?

Ma27 commented 2 years ago

You can use the ResolvConf= setting from systemd.nspawn(5). With that you can either forward the host's resolv.conf or not. The container-side uses networkd so you can configure your own networkd/resolved setup within the container as well.

The whole point of the change is to support as much of the usptream features as possible.

Btw, I know that the PR requires a bit of work, I'll be on vacation now, after that I'll hope that I'll get to work on that :)