nix-community / nixos-images

Automatically build (netboot) images for NixOS [maintainer=@Mic92]
MIT License
187 stars 35 forks source link

ipv4 route via ipv6 #230

Closed name-snrl closed 1 month ago

name-snrl commented 3 months ago

I tried to deploy NixOS using nixos-anywhere and ran into a problem. In my case, routing for ipv4 is configured via ipv6, it looks like this:

~/kexec# ./ip -4 route                   
default via inet6 fe80::1 dev ens10f0np0 

script checks that protocol is kernel and drops the route. Is there any solution for this?

Mic92 commented 3 months ago

Any idea what proto is actually set to if not specified? We can probably also pass through routes if proto is set to "static" or not set at all. The only thing we want to filter out is dhcp most of the time.

name-snrl commented 3 months ago

Any idea what proto is actually set to if not specified?

Oops, my bad, I said it was set to kernel but that was for docker. I think we just need to add an extra check for family?

ip -4 --json route | jq
[
  {
    "dst": "default",
    "via": {
      "family": "inet6",
      "host": "fe80::1"
    },
    "dev": "ens10f0np0",
    "flags": []
  },
  {
    "dst": "172.17.0.0/16",
    "dev": "docker0",
    "protocol": "kernel",
    "scope": "link",
    "prefsrc": "172.17.0.1",
    "flags": [
      "linkdown"
    ]
  }
]
Mic92 commented 3 months ago

Yes. But it needs to be mapped to the right networkd configuration.

name-snrl commented 3 months ago

BTW, to configure it in netwrokd, we have to set Destination= key. In my current machine config it looks like this:

[Route]
Destination=::/0
Gateway=fe80::1
GatewayOnLink=true

[Route]
Destination=0.0.0.0/0
Gateway=fe80::1
GatewayOnLink=true
Mic92 commented 3 months ago

Looks like we would than also need to parse "via", "host" here:

https://github.com/nix-community/nixos-images/blob/f267906ba610b2e29f3f45ab45a0669e28c407ef/nix/kexec-installer/restore_routes.py#L65

Just now we skip everything that has no Gateway. Could be also extended in our test suite to avoid future regressions: https://github.com/nix-community/nixos-images/blob/f267906ba610b2e29f3f45ab45a0669e28c407ef/nix/kexec-installer/test.nix#L155 Pull requests are welcome.

name-snrl commented 3 months ago

Pull requests are welcome.

At the moment I have already deployed one machine and am in the process of configuring it, but the plan is to deploy a cluster in the future, and if no one has fixed this by then - I will do so.