DNS leakage on Ubuntu 18.10.

[ghost]

I'm experiencing DNS leakage using this script in which the IP addresses of my ISP's DNS servers are visible. Strangely, the IP addresses of my VPN's DNS servers are also visible. I'm running an OpenVPN server connected to a Comcast router.

From DNS Leak Test:

IP	Hostname	ISP
208.67.219.70	m41.pao.opendns.com	OpenDNS, LLC
208.67.219.14	m4.pao.opendns.com	OpenDNS, LLC
76.96.15.73	sjos-cns05.nlb.sjc1.comcast.net	Comcast Cable
208.67.219.29	m21.pao.opendns.com	OpenDNS, LLC
...	...	...

systemd-resolve --status reports that everything is fine with the tunnel, so I assume I'm leaking traffic to some other interface:

Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 208.67.220.220
         DNS Servers: 208.67.222.222
                      208.67.220.220
          DNS Domain: ~.

Link 3 (wlp4s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 75.75.76.76
         DNS Servers: 75.75.75.75
                      75.75.76.76
                      2001:558:feed::1
                      2001:558:feed::2
          DNS Domain: ~.
                      hsd1.ca.comcast.net

Link 2 (enp0s31f6)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

OpenVPN client configuration:

...
script-security 2
setenv PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre

# Prevent DNS leakage.
dhcp-option DOMAIN-ROUTE .

I know very little about networking, so I apologize if this is a trivial issue. Any insight would be greatly appreciated.

[piotr-dobrogost]

I guess it is due to DNS Domain: ~. set on link wlp4s0 (wireless I guess). Is there any reason for this setting? If not then removing it should not break anything and prevent leakage here.

[ghost]

Thanks for the reply @piotr-dobrogost.

I'm not sure what the DNS Domain: ~. directive actually does, but it was referenced in this comment to specify that the VPN's DNS servers should take priority.

I don't know why DNS Domain: ~. appears in the wireless interface when the VPN is not running and when the VPN is run through the OpenVPN client using sudo openvpn --config ./client.ovpn. When run using NetworkManager, there is no such directive and no DNS leakage:

Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 14 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 208.67.222.222
         DNS Servers: 208.67.222.222
                      208.67.220.220
          DNS Domain: ~.

Link 3 (wlp4s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 75.75.75.75

This doesn't seem like a problem with update-systemd-resolved.

[jonathanio]

DNS Domain: ~. is the search configuration - especially systemd-resolved has been told that the servers configured on this link are responsible for searches on all domains (. is in effect the root domain, so all domains will match). Your tun0 interface has the same search query, so you will be sending out requests to via interfaces to both sets of DNS servers and the first successful response back will be the one provided to your application.

My current configuration is:

Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no
Fallback DNS Servers: 1.1.1.1
                      9.9.9.10
                      8.8.8.8
                      2606:4700:4700::1111
                      2620:fe::10
                      2001:4860:4860::8888
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (roadwarrior)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 3 (enp12s0u1u2)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 2001:4860:4860::8844
         DNS Servers: 8.8.8.8
                      8.8.4.4
                      2001:4860:4860::8844
                      2001:4860:4860::8888

Link 2 (wlp2s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 8.8.8.8
         DNS Servers: 8.8.8.8
                      8.8.4.4

As you can see, I do not have any DNS Domain configuration by default, so when I enable a VPN with DOMAIN-ROUTE . it will take over. Something on your system is overriding this configuration for some reason. What do you use to set up your networking configuration on your system?

[piotr-dobrogost]

I don't know why DNS Domain: ~. appears in the wireless interface when the VPN is not running and when the VPN is run through the OpenVPN client using sudo openvpn --config ./client.ovpn. When run using NetworkManager, there is no such directive and no DNS leakage:

That's interesting as it's Network Manager which likes to set ~. according to https://github.com/systemd/systemd/issues/10125

[ghost]

I haven't manually configured systemd-resolved in /etc/systemd/resolved.conf or any other place. file /etc/resolv.conf reports /etc/resolv.conf: symbolic link to ../run/systemd/resolve/stub-resolv.conf, so direct DNS requests should be routed to systemd-resolved's DNS stub. /etc/resolv.conf/ contains the following:

nameserver 127.0.0.53
search hsd1.ca.comcast.net

Thank you for explaining the purpose of the DNS Domain: ~. directive. I don't want to take any more of your time because this is unrelated to update-systemd-resolved. Is there a more appropriate place for this question, perhaps Networking Stack Exchange or systemd?

[jonathanio]

I have never been a fan of NetworkManager - much like systemd at times, it tries to do too much with a number of questionable decisions under the surface which often result in difficult to detect and resolve failure modes. There is, as in this example, absolutely no reason to demand for systemd-resolved.service that all DNS queries should be always directed to the local DNS server on the LAN when other, better, configurations may be provided.

Ultimately, and as you suspected, the reason why it works in your case with dhcp-option DOMAIN-ROUTE . on your VPN connection is that systemd, when faced with multiple possible options to route a query (explicit, or implicit), will send the same query to all servers. The result that is returned is either the first successful response (which can lead to some unpredictability without proper routing) or the last failed response (i.e. no server could resolve). With both links configured for ~. you'll probably always get one failure (LAN) and one success (VPN), so the success will be returned.

I'm uncertain how to remove ~. from your LAN interface, but ultimately this will help you if you can manage this configuration.

[bohlstry]

If have written a Bash-Script to get rid of ~. from all non-VPN-devices, if VPN is active. If VPN goes down, ~. is written to the default device (identified by the default route). Important: as mentioned above, dhcp-option DOMAIN-ROUTE . is required in VPN config.

Save the following code at: /etc/NetworkManager/dispatcher.d/10-check-dns-domain, don't forget to give execute rights.

#!/bin/bash

# Process only TUN (i.e. VPN) devices!

if [[ $1 == tun* ]]; then

  case $2 in

    up)
      logger -t "check-dns-domain" "TUN up: $1"
      # Remove from all but the current TUN interface the default domain search flag '~.'...
      for device in $(nmcli -t device | cut -d: -f1 | sort | grep -vx "$1"); do
        # Get current domain entries...
        domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
        # Check for an '~.' entry (surrounded by spaces or at begin/end of string)...
        if echo "$domain" | grep -qE '(^|\s)~\.(\s|$)'; then
          # Remove '~.', trim whitespace...
          domain=$(echo "$domain" | sed -E -e 's/(^|\s)~\.(\s|$)/ /g' -e 's/^\s+//' -e 's/\s+$//')
          logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
          if [ -z "$domain" ]; then
            # Domain is now empty...
            resolvectl domain "$device" ""
          else
            # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
            resolvectl domain "$device" $domain
          fi
        fi
      done
      ;;

    down)
      logger -t "check-dns-domain" "TUN down: $1"
      # Get device name of default route...
      device=$(ip route ls | perl -ne 'do {print $1; last} if /^default\s.*\sdev\s(\S+)/')
      # Get current domain entries...
      domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
      # Append '~.', separated by space. If domain was empty, just set value (without space)...
      domain=${domain:+$domain }~.
      logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
      # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
      resolvectl domain "$device" $domain
      ;;

  esac

fi

To check the log, you can use journalctl -f -t check-dns-domain

[dannyk81]

Thanks @bohlstry for sharing the script, works great for me.

I tried every possible way to get rid of ~. from non-VPN (Wifi) device but it keeps coming back, I suppose by NetworkManager (?)

Using this method seems like the only way to fix DNS leakage...

EDIT: The script handles the case of a single VPN well, however I'm normally connected to multiple VPNs, so when I disconnected from just one it pushes back ~.

EDIT 2: slightly revised version of @bohlstry's script, works in case there are multiple VPN connections:

#!/bin/bash

function remove_default_search_flag() {
  # Remove from all but the current TUN interface the default domain search flag '~.'...
  for device in $(nmcli -t device | cut -d: -f1 | sort | grep -vx "$1"); do
    # Get current domain entries...
    domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
    # Check for an '~.' entry (surrounded by spaces or at begin/end of string)...
    if echo "$domain" | grep -qE '(^|\s)~\.(\s|$)'; then
      # Remove '~.', trim whitespace...
      domain=$(echo "$domain" | sed -E -e 's/(^|\s)~\.(\s|$)/ /g' -e 's/^\s+//' -e 's/\s+$//')
      logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
      if [ -z "$domain" ]; then
        # Domain is now empty...
        resolvectl domain "$device" ""
      else
        # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
        resolvectl domain "$device" $domain
      fi
    fi
  done
}

function add_default_search_flag() {
  # Get device name of default route...
  device=$(ip route ls | perl -ne 'do {print $1; last} if /^default\s.*\sdev\s(\S+)/')
  # Get current domain entries...
  domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
  # Append '~.', separated by space. If domain was empty, just set value (without space)...
  domain=${domain:+$domain }~.
  logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
  # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
  resolvectl domain "$device" $domain
}

# Process only TUN (i.e. VPN) devices!
if [[ $1 == tun* ]]; then

  case $2 in

    up)
      logger -t "check-dns-domain" "TUN up: $1"
      remove_default_search_flag "$1"
      ;;

    down)
      logger -t "check-dns-domain" "TUN down: $1"
      # Check if there are other active TUN devices
      active_tuns=$(nmcli connection show --active|grep -q tun && echo 0 || echo 1)
      if [ $active_tuns -eq 0 ]; then
        logger -t "check-dns-domain" "Other TUN devices are active"
        remove_default_search_flag "$1"
      else
        logger -t "check-dns-domain" "No additional active TUN devices"
        add_default_search_flag
      fi
      ;;

  esac

fi

[Edu4rdSHL]

Hello, you just need to use the following option in your .network file:

[DHCP]
UseDNS=no

The "leakage" that you're getting is because if you don't specify that option, the DNS servers received from the DHCP server will be used and take precedence over any statically configured ones.

See https://www.freedesktop.org/software/systemd/man/systemd.network.html#%5BDHCP%5D%20Section%20Options

[jonathanio]

@bohlstry and @dannyk81,

Thank you both for your scripts. It's still a shame that NetworkManager has not looked into this and doesn't seem to be any closer to resolving this regression. I'm much more a fan of systemd-networkd, despite even its issues at times.

Nonetheless, I'll keep this open waiting for a resolution from NetworkManager. In the meantime, I've added into the documentation notes about the issue of leakage when using NetworkManager, and linked to this issue.

[Edu4rdSHL]

I don't understand what have to be network-manager in a script for systemd-resolved... network-manager provides their own DNS resolver implementation, nothing to be with systemd-resolved.

[vaskokj]

How do I go about removing the DNS Domain: ~. from my primary connection? I have no idea how that is even getting there in the first place? Is this something that my DHCP server adding this? If not, how do I go about removing it?

[nijave]

How do I go about removing the DNS Domain: ~. from my primary connection? I have no idea how that is even getting there in the first place? Is this something that my DHCP server adding this? If not, how do I go about removing it?

systemd-resolve --status or resolvectl status depending on what version of systemd you have. Look for the Link # in the output with DNS Domain set

Link 3 (br-63697f7718b1)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (wlp59s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 1.1.1.1
          DNS Domain: ~.

i.e. link 2 here

Run

sudo busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDomains 'ia(sb)' <link id #> 0

The 0 sets it back to nothing which effectively removes any entries

[DemiMarie]

Have the NetworkManager bugs been fixed? Another approach is to use nftables rules to block all non-VPN traffic. @jonathanio what do you think of this option?

[coderonline]

I have had the same issue with a Fritz!box, because these seem to push DNS settings via DHCP and that is where the ~. DNS Domain came from in my case. It makes systemd-resolved send DNS queries out on both network-interfaces and takes the first to answer. That is the DNS leak I wanted to get rid of.

I was able to track this further down and found in the Arch-Wiki a way to disable Domains from the Fritz!Box altogether (this may not be what you want, but for me its okay, because I can still use host something.fritz.box 192.168.178.1 to find devices in my local network). Then I had trouble to start VPN and found this issue with a working solution for the second issue "Call failed: Link tun0 is managed.".

With the following two configuration files I got it working for me :)

How to get it working

On my client I have now two .network-files for systemd:

1. /etc/systemd/network/enp8s0.network

   
   [Match]
   Name=enp8s0

[DHCP] UseDomains=no

2. `/etc/systemd/network/tun.network`:

[Match] Name=tun*

[Link] Unmanaged=yes

[Network] UseDomains=yes

and then restarted `systemctl restart systemd-networkd` and `systemctl restart systemd-resolved`, reconnected to my network and finally restarted my VPN with `openvpn-client@someserver` and now have no `~.` on enp8s0 any more and that is the best solution, I think.

## FYI: "how to remove ~. from a specific interface"

With the configuration above `resolvectl domain enp8s0 fritz.box ~.` would re-add the unwanted configuration temporarily and `resolvectl domain enp8s0 fritz.box` would remove only the `~.`

[tmccombs]

Wouldn't removing "~." from my hardware interface, cause DNS resolution to fail (or at least not use those resolvers) if the VPN isn't active? Is there a way to remove the ~. from other interfaces only while the vpn is active? or at least tell systemd-resolved to try the dns resolvers for tun0 before any other interfaces?

[tmccombs]

It also looks like update-systemd-resolved doesn't call SetDefaultRoute. Should there be some way to configure this to set that?

[piotr-dobrogost]

Users of NetworkManager might be interested in the recent fix regarding this issue – dns: fix handling default routing domains with systemd-resolved (https://github.com/freedesktop/NetworkManager/commit/ee9fab03613e30f818d56217a968d1fabd5ea8d7)

[jan-swiecki]

Users of NetworkManager might be interested in the recent fix regarding this issue – dns: fix handling default routing domains with systemd-resolved (freedesktop/NetworkManager@ee9fab0)

Any idea how to install newest NM (network manager) on Ubuntu. I'm using 20.04 LTS, but maybe there is a universal way to do it? It would seem that I need to build it from source and somehow seamlessly replace apt-installed system NM with newly built version, but I'm unsure how to do replacing step.

[tomeon]

@tmccombs

It also looks like update-systemd-resolved doesn't call SetDefaultRoute. Should there be some way to configure this to set that?

Please see #110 for support for (among other things) controlling SetLinkDefaultRoute.

[tomeon]

Given https://github.com/jonathanio/update-systemd-resolved/issues/59#issuecomment-732436761, is it time to close this issue?

[tomeon]

The NetworkManager fix mentioned above landed in NetworkManager release 1.26.6.

[tomeon]

Also, AFAICT, the latest Ubuntu LTS ships a version of NetworkManager that contains the fix:

$ cat /etc/issue
Ubuntu 22.04.3 LTS \n \l

$ apt-cache policy network-manager
network-manager:
  Installed: (none)
  Candidate: 1.36.6-0ubuntu2
  Version table:
     1.36.6-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.36.4-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages