VMs depending on a VPN ProxyVM cannot reach the DNS server

[ramboman]

Qubes OS release

4.2.3

Brief summary

Since I upgraded my VPN ProxyVM (sys-vpn) from fedora-39-xfce to fedora-40-xfce, its depending VMs cannot reach the DNS server anymore.

Steps to reproduce

• dom0: Install fedora-40-xfce VM
• dom0: Update fedora-40-xfce VM
• dom0: Create the ProxyVM:
  • Name: sys-vpn
  • Type: AppVM
  • Template: fedora-40-xfce
  • Networking: sys-firewall
  • provides network: true
  • Services:
  • network-manager
• dom0: Create the ProxyVM dependent VM:
  • Name: test-vpn
  • Type: DispVM
  • Template: Some AppVM
  • Networking: sys-vpn
• dom0: Start sys-vpn VM
• sys-vpn: Setup VPN according to NetworkManager documentation:
  • In my case:
  • Left-click on the NetworkManager Systray Icon
  • VPN Connections -> Configure VPN...
  • Click on the + button
  • Choose Import a saved VPN configuration...
  • Add username and password to the configuration
• sys-vpn: Check nft

Result:

$ sudo nft list chain ip qubes dnat-dns
table ip qubes {
    chain dnat-dns {
        type nat hook prerouting priority dstnat; policy accept;
        ip daddr 10.139.1.1 udp dport 53 dnat to 10.139.1.1
        ip daddr 10.139.1.1 tcp dport 53 dnat to 10.139.1.1
        ip daddr 10.139.1.2 udp dport 53 dnat to 10.139.1.2
        ip daddr 10.139.1.2 tcp dport 53 dnat to 10.139.1.2
    }
}

• sys-vpn: Start VPN
• sys-vpn: Check nft

Expected behavior

Result:

$ sudo nft list chain ip qubes dnat-dns
table ip qubes {
    chain dnat-dns {
        type nat hook prerouting priority dstnat; policy accept;
        ip daddr 10.139.1.1 udp dport 53 dnat to 1.1.1.1
        ip daddr 10.139.1.1 tcp dport 53 dnat to 1.1.1.1
        ip daddr 10.139.1.2 udp dport 53 dnat to 8.8.8.8
        ip daddr 10.139.1.2 tcp dport 53 dnat to 8.8.8.8
    }
}

sys-vpn reroute correctly the DNS request to the proper DNS server

• dom0: Start test-vpn VM
• test-vpn: ping www.google.com + ctrl+c

Result:

$ ping www.google.com
PING www.google.com (64.233.177.99) 56(84) bytes of data.
64 bytes from yx-in-f99.1e100.net (64.233.177.99): icmp_seq=2 ttl=55 time=36.1 ms
64 bytes from yx-in-f99.1e100.net (64.233.177.99): icmp_seq=3 ttl=55 time=35.4 ms
64 bytes from yx-in-f99.1e100.net (64.233.177.99): icmp_seq=4 ttl=55 time=35.9 ms
^C
--- www.google.com ping statistics ---
4 packets transmitted, 3 received, 25% packet loss, time 3033ms
rtt min/avg/max/mdev = 35.380/35.790/36.111/0.305 ms

test-vpn is able to reach the DNS server and Google.

Actual behavior

Result:

$ sudo nft list chain ip qubes dnat-dns
table ip qubes {
    chain dnat-dns {
        type nat hook prerouting priority dstnat; policy accept;
        ip daddr 10.139.1.1 udp dport 53 dnat to 10.139.1.1
        ip daddr 10.139.1.1 tcp dport 53 dnat to 10.139.1.1
        ip daddr 10.139.1.2 udp dport 53 dnat to 10.139.1.2
        ip daddr 10.139.1.2 tcp dport 53 dnat to 10.139.1.2
    }
}

sys-vpn does not reroute correctly the DNS request to the proper DNS server.

• dom0: Start test-vpn VM
• test-vpn: ping www.google.com + ctrl+c

Result:

$ ping www.google.com
<nothing>

test-vpn is unable to reach the DNS server and Google.

Further details

• I use the NetworkManager systray icon to start the VPN.
• NetworkManager uses a hook script (/etc/NetworkManager/dispatcher.d/qubes-nmhook) to reroute the DNS when the right DNS server.
• qubes-nmhook relies on /usr/lib/qubes/qubes-setup-dnat-to-ns to reroute the DNS.
• After I start the VPN, in qubes-setup-dnat-to-ns, get_dns_resolved(), in my case:
  • in fedora-39-xfce, returns: [1.1.1.1, 8.8.8.8, 10.139.1.1, 10.139.1.2, 1.1.1.1, 8.8.8.8]
  • in fedora-40-xfce, returns: [10.139.1.1, 10.139.1.2, 1.1.1.1, 8.8.8.8]
• In qubes-setup-dnat-to-ns, install_firewall_rules() uses the first 2 elements of the dns_resolved list to generate the nft rules.
• Before starting this issue, I made a post on the forum about this subject here. apparatus seems to have found a solution.