netmasks are interpreted wrongly

[edef1c]

When /etc/network/interfaces specifies a /18, the network setup script interprets it as being a /16:

# cat /etc/network/interfaces 
…
auto eth0
iface eth0 inet static
        address …
        netmask 255.255.192.0
        gateway …
        up ip addr add … dev eth0
        dns-nameservers 8.8.8.8 8.8.4.4

# ip a
…
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether … brd ff:ff:ff:ff:ff:ff
    inet …/16 scope global eth0
       valid_lft forever preferred_lft forever
…

[jeaye]

Unfortunately, you've removed some useful information from your snippet. The up ip addr add line contains the bit which nixos-in-place reads as the prefix length.

The script, here, mentions this example data:

iface eth0 inet static
        address 188.166.222.72
        netmask 255.255.240.0
        gateway 188.166.208.1
        up ip addr add 10.15.0.7/16 dev eth0
        dns-nameservers 8.8.8.8 8.8.4.4

Will you share the relevant prefix length in your /etc/network/interfaces and what your expectations are, based on that data? If you'd like to change the ips to xx.xx.xx.xx, that's fine, but the rest of the information is crucial.

[edef1c]

@jeaye That prefix length is for the anchor address, not for the main address. The anchor address is only for floating IPs, not for the primary address. The netmask listed with the address is (surprise!) the actual netmask of the address.

[edef1c]

It seems ip(1) doesn't take a netmask, only a prefix length as part of a CIDR, so I'm not sure how we could fix this in bash. (I settled on a small program that configures systemd-networkd based on the information from the Digital Ocean metadata service myself)

[jeaye]

@nathan7 Would you mind sharing your solution? I'm open to improving this and it seems like you have more experience in this area than me.

[edef1c]

@jeaye https://code.nathan7.eu/nathan7/systemd-digitalocean This is my solution, which currently runs on several of my servers (I converted them to pure NixOS after using nixos-in-place — so many thanks for that!) It's missing the hardware configuration (I'm hoping to PR that into nixpkgs separately soon), but I'm unsure that my Go program will be cool with the nixpkgs maintainers.

[cstrahan]

@nathan7 We're already using a small Go program for the dockerTools functions, so I would feel comfortable with your approach using Go. A PR for better DO support would be greatly appreciated.

It's missing the hardware configuration

What sort of hardware configuration are you referring to here?

[edef1c]

@cstrahan Cool, though I'm not sure sticking Go source in the nixpkgs tree like with dockerTools would be appropriate here. I'm planning to do a little more work on this before I PR it to nixpkgs, but that is my long-term plan.

It's missing the hardware configuration

What sort of hardware configuration are you referring to here?

https://code.nathan7.eu/nathan7/colossus.nathan7.eu/src/f22d35d7458cf088ed25ee08c30611c19f783b32/digitalocean/default.nix, analogous to <nixpkgs/nixos/modules/virtualisation/amazon-image.nix>

[KibaFox]

I installed NixOS on a Digital Ocean droplet. I started with a Debian 64-bit base. After installing, I couldn't get my machine to see the internet.

It looks to me like my static IPv4 address isn't getting set correctly. As a note, I enabled IPv6 on this droplet. Is this the same issue?

Here's what I have running the same commands:

[jeaye]

The issue here is that the existing setup-network script is simply not robust enough to handle anything but the most trivial cases. It doesn't handle ipv6 currently either. The primary reason for this is my lack of knowledge in this area; it'd be nice to get some more eyes on it and develop some improvements. Ideally, we stick to bash and not Go, Rust, C, or anything of the sort.

[KibaFox]

Thank you. I'm not very familiar with this area either, but I think a bash solution would be ideal here as well.

[KibaFox]

Is there a reason why we're not converting the network settings to a nix expression in the NixOS configuration?

Maybe we could generate something like the following when we build nixos-in-place.nix...

{
  networking = {
    usePredictableInterfaceNames = false;  # Use eth0, eth1, etc
    interfaces = {
      eth0.ip4 = [{
        address = "162.243.123.70";
        prefixLength = 24;
      }];
      defaultGateway = "162.243.123.1";
    };
  };
}

The snippet is missing Ipv6 settings, but this is just to give you an idea.

[edef1c]

@jeaye If you're hoping to do netmask manipulation from bash, I applaud your bravery, but that's probably where I split off and aim for the practical instead. Reading data from the metadata service at runtime is much more robust than parsing the /etc/network/interfaces file (but requires HTTP and optionally JSON manipulation), and having systemd-networkd handle the actual interface configuration meshes more easily with the rest of the system (this is what I do)

[KibaFox]

(I converted them to pure NixOS after using nixos-in-place...

@nathan7 How did you convert to pure NixOS?

[edef1c]

@KibaFox Deleted everything belonging to the original OS, rebuilt without the nixos-in-place extras, copied the Nix store from /nixos/nix to /nix (with rsync), and rebooted. Some care required.

[KibaFox]

@nathan7 Thanks! Do you know if there is anything wrong with setting the networking configuration as part of the NixOS configuration (in /etc/nixos/configuration.nix)?

Is there a reason we're parsing /etc/network/interfaces? Does Digital Ocean edit it?

[edef1c]

@KibaFox DO used to write to it as the primary way of configuring the network, but none of the current images make use of that — you're expected to get this information from the link-local metadata service. It's written to when you create a new droplet — if we hardcode it in configuration.nix, every image would have to be provisioned by hand, like in an absurd alternate universe where DHCP was never invented.

[KibaFox]

I see. Thank you for explaining that for me, @nathan7.

[rimmington]

@nathan7 Are you still working on systemd-digitalocean?

(I don't appear to be able to open issues at code.nathan7.eu, sorry for hijacking this issue a bit)

[edef1c]

@rimmington I'm still using it, and intend to keep doing so — I don't really expect upstreaming it into NixOS to be all that much fun to bikeshed over, so I've been holding off on that.