Nftables firewall blocks requests to dnsmasq DHCP

[FlyingWombat]

When networking.nftables.enable = true; libvirt and lxd guests fail to run DHCP request with dnsmasq on host. Work-around is to set networking.firewall.trustedInterfaces = [ "lxdbr0" "virbr0" ]; Similar to #10101 but setting networking.firewall.checkReversePath = false; doesn't fix the problem.

Setting trustedInterfaces isn't ideal, because it removes a layer of separation (the firewall) between the guests and the host.

Is there a better way to fix this?

EDIT: with networking.nftables = false; I run into this issue only with LXD, not libvirt. But with true it happens for both.

Steps To Reproduce

Set networking.nftables.enable = true; and networking.firewall.enable = true; , nixos-rebuild, restart libvirtd.service, and start a guest.

Expected behavior

guests can send DHCP requests to host dnsmasq via virtual network bridge.

Notify maintainers

@mkg20001

Metadata

nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.5.8, NixOS, 23.11 (Tapir), 23.11.20231023.b3bc9d1`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.1`
 - channels(root): `"nixos"`
 - nixpkgs: `/run/current-system/nixpkgs`

[mkg20001]

I'm not sure how it worked with iptables before

What's possible is to add extra trusted interfaces at runtime via a set, but im not sure if thats the best solution

If its possible and the software affected already does nftables rules management it could accept the packets before the nixos nftables rules reject it

(which would also mean changing the priority of the nixos-fw table hooks to be after the default priority for each hook)

[duament]

You may use the networking.firewall.extraInputRules option like this:

networking.firewall.extraInputRules = ''
  iifname { "lxdbr0", "virbr0" } meta l4proto { tcp, udp } th dport 53 accept comment "DNS"
  iifname { "lxdbr0", "virbr0" } meta nfproto ipv4 udp dport 67 accept comment "DHCP server"
'';

[khaled]

I'm not enabling/disabling nftables explicitly, but I started running into this issue after updating my system from a nixpkgs-unstable rev from around 8/30 to one from 9/15. The workaround of setting networking.firewall.trustedInterfaces = [ "lxdbr0" "virbr0" ]; worked for me.

It seems that I am using nftables, since iptables --version yields iptables v1.8.9 (nf_tables).

But perplexingly, nixosConfigurations.myhost.config.networking.nftables.enable evaluates to false in nix repl. Not sure what that means.

[FlyingWombat]

@khaled Instead of networking.firewall.trustedInterfaces = [ "lxdbr0" "virbr0" ]; I've recently found that it also works with allowing only the dnsmasq ports 53 and 67, similar to how @duament suggested.

  networking.firewall.interfaces.lxdbr0.allowedTCPPorts = [ 53 ];
  networking.firewall.interfaces.virbr0.allowedTCPPorts = [ 53 ];
  networking.firewall.interfaces.lxdbr0.allowedUDPPorts = [ 53 67 ];
  networking.firewall.interfaces.virbr0.allowedUDPPorts = [ 53 67 ];

I think, before this issue can be closed, we need to discuss how we can fix this automatically, instead of users unwittingly breaking their guest networking.

But perplexingly, nixosConfigurations.myhost.config.networking.nftables.enable evaluates to false in nix repl.

False is the default value. Someone more knowledgeable may have to correct me, but Nixos some time ago switched to the iptables-nft backend, which is why iptables is reporting that. The networking.nftables option (I'm assuming) configures Nixos firewall rules directly with nftables, instead of using the legacy iptables interface (with nftables backend), and changes the systemd firewall unit to nftables.service instead of firewall.service.

Turns out, with networking.nftables = false; I run into this issue only with LXD, not libvirt, but with true it happens for both. Are you seeing the issue with LXD or libvirt or both?

[khaled]

@FlyingWombat thanks for the informative note + explanation! Agreed that this should probably not be closed yet. I'm wondering why the behavior seemed to have changed between nixpkgs revisions. Previously I did not have to set trusted interfaces or open any ports.

I had only run into this issue with LXD, but just now tried libvirt after setting network.nftables = true; and have the same issue there too.

[htngr]

@khaled Instead of networking.firewall.trustedInterfaces = [ "lxdbr0" "virbr0" ]; I've recently found that it also works with allowing only the dnsmasq ports 53 and 67, similar to how @duament suggested.

  networking.firewall.interfaces.lxdbr0.allowedTCPPorts = [ 53 ];
  networking.firewall.interfaces.virbr0.allowedTCPPorts = [ 53 ];
  networking.firewall.interfaces.lxdbr0.allowedUDPPorts = [ 53 67 ];
  networking.firewall.interfaces.virbr0.allowedUDPPorts = [ 53 67 ];

This solved #255110 for me.

[hmenke]

The proposed workarounds only apply to the default network. As soon as a new network is created, it receives a new interface name which then has to be added manually to configuration.nix. A better solution is clearly needed.

[folliehiyuki]

nftables allows wildcards for iifname and oifname, so you can do something like this in input chain: iifname "virbr*" udp dport { 53, 67 } accept.

Judging by the implementation at https://github.com/NixOS/nixpkgs/blob/3030f185ba6a4bf4f18b87f345f104e6a6961f34/nixos/modules/services/networking/firewall-nftables.nix#L125, I think we can just pass the expression into Nix configuration and it hopefully can work (I haven't tested yet):

  networking.firewall.interfaces."virbr*".allowedUDPPorts = [ 53 67 ];

Ref: https://manpages.debian.org/bullseye/nftables/nft.8.en.html#META_EXPRESSIONS

[hmenke]

@folliehiyuki That's slightly better but still requires that every new interface be named virbr*. Instead of statically configuring the firewall, libvirt should use appropriate hooks to configure the firewall dynamically as new networks are added. https://libvirt.org/hooks.html#etc-libvirt-hooks-network

[jtojnar]

Looks like the same change also broke NetworkManager Wi-Fi hotspot. Without DHCP port open on firewall, my Android phone will get stuck on “Obtaining IP address” and without DNS, it will report not being connected to the Internet.

Not sure if it is good idea to enable the ports on the wlp2s0 interface used for hotspot, as the same interface is other times used for connecting to public Wi-Fi networks.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/wifi-sharing-gnome-and-kde-plasma/44195/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/simple-standalone-network-manager-wifi-hotspot/49327/2