Container private networking not working when hardened profile is enabled

[andrewchambers]

Issue description

Container start fails with privateNetwork = true; Does start when sharing host network.

Steps to reproduce

        containers.gitlab =
          { config =
              { config, pkgs, ... }:
              { services.openssh.enable = true;
                services.openssh.ports = [ 3344 ];
              };
                privateNetwork = true;
                hostAddress = "192.168.100.10";
                localAddress = "192.168.100.11";
          };

$ sudo nixos-container start gitlab
Job for container@gitlab.service failed because the control process exited with error code.
See "systemctl status container@gitlab.service" and "journalctl -xe" for details.
/run/current-system/sw/bin/nixos-container: failed to start container
$ systemctl status container@gitlab.service
Failed to add new veth interfaces (ve-gitlab:host0): Operation not supported
... failed

Technical details

 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.32, NixOS, 18.03.131807.489a14add9a (Impala)`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0`
 - channels(root): `"nixos-18.03.131807.489a14add9a"`
 - channels(ac): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs`

[xeji]

I cannot reproduce this on my system with same nixos revision 18.03.131807.489a14add9a and your container config from above. Container starts and networking looks fine. Does your system have any kind of special network setup?

[andrewchambers]

Whole config below :

Only things that look interesting to me are virtualbox and perhaps the rkt service.

# Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:
let
    acpkgs = (import (pkgs.fetchFromGitHub {
       owner = "andrewchambers";
       repo = "acpkgs";
       rev = "12dc2f7787249de3e0a5cab6dc74f50ce168ceb1";
       sha256 = "07g7mknbgg1gdv8bg1jjsi5gw8zzi1dq9jzwrgpqmzg3qmddczbj";
     })) {pkgs = pkgs;};
    hashedPassword = import ./password.nix;
    systemBackup = pkgs.writeScriptBin "systembackup.sh" ''
        #! /bin/sh

        set -e
        set -u

        if test -e /rootsnap
        then
            echo "removing snapshot, it already existed."
            btrfs subvolume delete /rootsnap
        fi

        btrfs subvolume snapshot -r / /rootsnap > /dev/null

        export RESTIC_PASSWORD_FILE=/root/backups_password
        export RESTIC_REPOSITORY=sftp://backups@acha.ninja//backups/restic

        restic backup --quiet --exclude /rootsnap/home/ac/Downloads --exclude /rootsnap/home/ac/.cache /rootsnap/etc/nixos/ /rootsnap/home/
        ssh backups@acha.ninja sync

        btrfs subvolume delete /rootsnap > /dev/null
    '';
in
    {
        imports =
        [ # Include the results of the hardware scan.
          ./hardware-configuration.nix
          <nixpkgs/nixos/modules/profiles/hardened.nix>
        ];

        nix.allowedUsers = ["@wheel"];

        sound.enable = true;
        hardware.pulseaudio.enable = true;
        hardware.pulseaudio.systemWide = true;

        boot.supportedFilesystems = [ "btrfs" "ext4" ];
        boot.loader.systemd-boot.enable = true;
        boot.loader.efi.canTouchEfiVariables = true;
        boot.initrd.kernelModules = [ "dm-snapshot" ];
        boot.initrd.luks.devices = [
            {
                name = "root";
                device = "/dev/disk/by-uuid/658af668-4628-46e8-842a-cef4b31d55a1";
                preLVM = true;
            }
        ];
        boot.cleanTmpDir = true;
        boot.tmpOnTmpfs = true;

        nixpkgs.config.allowUnfree = true;

        networking.hostName = "black";

        networking.defaultMailServer = {
            directDelivery = true;
            hostName = "smtp.mailgun.org";
            domain = "home.acha.ninja";
            authUser = "XXX";
            authPass = "XXX";
            root = "andrewchamberss@gmail.com";
            useTLS = true;
            useSTARTTLS = true;
        };

        time.timeZone = "NZ";

        environment.systemPackages = with pkgs; [
            acpkgs.fifolog
            acpkgs.google-cloud-sdk
            acpkgs.kvdb
            acpkgs.qlite

            alsaUtils
            terraform_0_11-full
            systemBackup
            archivemount
            sshfs
            bup
            clang
            curl
            daemontools
            dep
            lsof
            direnv
            erlang
            firefox
            gcc
            git
            gnupg
            go
            jq
            jo
            nixops
            minikube
            mkpasswd
            nix-prefetch-git
            ntfs3g
            pinta
            python2
            python3
            restic
            rzip
            sublime3
            transmission
            unzip
            vim
            virtmanager
            vlc
            vscode
            wget
            xclip
            xdotool
            pass
            virtualbox
        ];

        services.openssh.enable = true;
        services.openssh.passwordAuthentication = false;

        services.xserver.enable = true;
        services.xserver.windowManager.i3.enable = true;

        services.cron.mailto = "root";
        services.cron.systemCronJobs = [
            "0 0,12 * * * root ${systemBackup}/bin/systembackup.sh"
        ];

        services.ipfs.enable = true;
        services.ipfs.gatewayAddress = "/ip4/127.0.0.1/tcp/3544";

        virtualisation.virtualbox.host.enable = true;
        virtualisation.rkt.enable = true;

        users.mutableUsers = false;
        users.extraUsers.ac = {
            isNormalUser = true;
            home = "/home/ac";
            description = "Andrew Chambers";
            extraGroups = [ "wheel" "libvirtd" "audio" ];
            hashedPassword = hashedPassword;
        };

        # This value determines the NixOS release with which your system is to be
        # compatible, in order to avoid breaking some software such as database
        # servers. You should change this only after NixOS release notes say you
        # should.
        system.stateVersion = "18.03"; # Did you read the comment?
    }

[andrewchambers]

[ac@black:~]$ ifconfig
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.3  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::127b:44ff:fe92:79a4  prefixlen 64  scopeid 0x20<link>
        inet6 2407:7000:8148:c471:127b:44ff:fe92:79a4  prefixlen 64  scopeid 0x0<global>
        inet6 fda4:7174:22fd:2c00:127b:44ff:fe92:79a4  prefixlen 64  scopeid 0x0<global>
        ether 10:7b:44:92:79:a4  txqueuelen 1000  (Ethernet)
        RX packets 14186941  bytes 3572625118 (3.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12543143  bytes 3092111475 (2.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xefc00000-efc1ffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 60484  bytes 5544865 (5.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 60484  bytes 5544865 (5.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vboxnet0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.56.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 0a:00:27:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:6d:66:06  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[andrewchambers]

Ok, found the cause, was the hardened profile. Perhaps bpf is needed for virtual ethernet, will investigate some more.

[xeji]

Thanks. I never tried containers with the hardened profile. Will look into it.

[xeji]

Don't understand why yet, but adding security.lockKernelModules = false; to the host config seems to fix it. Hardened profile sets this to true by default.

@andrewchambers please check if this works for you.

[andrewchambers]

@xeji confirmed after a reboot.

[xeji]

I think a better solution will be to have the container module load all kernel modules required for private networking at boot time. Let's keep the issue open while I give that a try.

[baracoder]

@xeji ping ;)

[joachifm]

Module locking is quite heavy-handed. Loading all required modules before the lock is engaged is indeed the correct fix, if locking is desired.

[yvt]

Adding boot.kernelModules = [ "veth" ]; seems to fix the issue. I see that at least start and root-login is working, but am not sure about other things.

[joachifm]

@yvt makes sense to me. If you care to, I'd be happy to integrate a PR to that effect.

[minijackson]

Hello ! I stumbled upon this issue while trying to add a container with a private network, so I'm not sure this issue is resolved: I still had to add boot.kernelModules = [ "veth" ]; as @yvt suggested, which fixed the container startup.

[serpent213]

Hello ! I stumbled upon this issue while trying to add a container with a private network, so I'm not sure this issue is resolved: I still had to add boot.kernelModules = [ "veth" ]; as @yvt suggested, which fixed the container startup.

Same here, still...