search

tailscale / tailscale

The easiest, most secure way to use WireGuard and 2FA.
https://tailscale.com
BSD 3-Clause "New" or "Revised" License
19.48k stars 1.52k forks source link

`/etc/resolv.conf` overwritten even though `systemd-resolved` is running and symlink present #11342

Open dseravalli opened 8 months ago

dseravalli commented 8 months ago

What is the issue?

Tailscale is overwriting /etc/resolv.conf when it should be working with systemd-resolved

Steps to reproduce

sudo systemctl stop tailscaled.service
sudo  ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
cat /etc/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 2606:4700:4700::1111
# Too many DNS servers configured, the following entries may be ignored.
nameserver 2606:4700:4700::1001
search .
systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-03-06 01:17:37 EST; 1min 19s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 6532 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 4416)
     Memory: 2.4M
        CPU: 265ms
     CGroup: /system.slice/systemd-resolved.service
             └─6532 /lib/systemd/systemd-resolved

Mar 06 01:17:37 raspberrypi systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Mar 06 01:17:37 raspberrypi systemd-resolved[6532]: Positive Trust Anchors:
Mar 06 01:17:37 raspberrypi systemd-resolved[6532]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Mar 06 01:17:37 raspberrypi systemd-resolved[6532]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.1>
Mar 06 01:17:37 raspberrypi systemd-resolved[6532]: Using system hostname 'raspberrypi'.
Mar 06 01:17:37 raspberrypi systemd[1]: Started systemd-resolved.service - Network Name Resolution.
sudo systemctl start tailscaled.service
sudo tailscale up
cat /etc/resolv.conf
# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search [redacted]

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu 23.04

Tailscale version

1.60.1

Other software

Bug report

BUG-8aee7c32b875212ae9f34e31c49cf202b3a2f1920d7ce7bf634ea06eba9b42da-20240306062104Z-3f3b9cad4e90b6f4

knyar commented 8 months ago

Looking at the logs from the bugreport provided:

2024-03-06 06:19:42.977814524 +0000 UTC: dns: resolvedIsActuallyResolver error: resolv.conf doesn't point to systemd-resolved; points to [1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001]
2024-03-06 06:19:42.977925744 +0000 UTC: dns: [resolved-ping=yes rc=resolved resolved=not-in-use ret=direct]

Relevant code is here: https://github.com/tailscale/tailscale/blob/0cb86468cab1b74b10589e94568620d31a22c9a6/net/dns/manager_linux.go#L338

Do you happen to know how 1.1.1.1 ended up being configured in stub-resolv.conf? I believe it's expected to have a single nameserver pointing to systemd-resolved address (127.0.0.53), and the comment in the beginning of that file warns against changing it.

dseravalli commented 8 months ago

Seems to be because I have cloudflare set up in /etc/systemd/resolved.conf. Is that incorrect? (note i removed the ipv6 addresses since original post)

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
DNS=1.1.1.1 1.0.0.1
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
DNSStubListener=no
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
dseravalli commented 8 months ago

Changing DNSStubListener=yes fixed it but now I'm unsure if 1.1.1.1 is being using for resolution.

JeffPaine commented 4 months ago

I think I see a path to this wonky state:

From https://tailscale.com/kb/1235/resolv-conf:

Tailscale overwrites /etc/resolv.conf when MagicDNS is enabled on the tailnet and --accept-dns is enabled on the machine running Tailscale and there doesn't appear to be a DNS manager running on the system. [emphasis mine]

I believe this logic is used to determine if systemd-resolved is managing resolv.conf, specifically this code that requires an entry like nameserver 127.0.0.53 (the address of the systemd-resolved stub resolver) in resolv.conf.

However, it appears here that /etc/resolv.conf was symlinked to a file with no such nameserver 127.0.0.53 line, thus Tailscale believed the file was not managed by systemd-resolved and thus overwrote it.

The check for nameserver 127.0.0.53 was added in https://github.com/tailscale/tailscale/commit/320cc8fa2191b1d04bcc54fd605db2d6d683405f, but I believe it has a logical flaw, from the description:

if the header comment names systemd-resolved, there should be a single nameserver in resolv.conf pointing to 127.0.0.53. If not, the configuration should be treated as an unmanaged resolv.conf.

It's possible for resolv.conf to be managed by systemd-resolved but not have a nameserver 127.0.0.53 line. For example, when /etc/resolv.conf is symlinked to /run/systemd/resolve/resolv.conf: the file is not expected to have a line like nameserver 127.0.0.53 in it (as the stub resolver is not enabled), but the config is nonetheless managed by systemd-resolved.

The current logic in manager_linux.go does work, however, when /etc/resolv.conf is symlinked to /run/systemd/resolve/stub-resolv.conf as the file should have a line like nameserver 127.0.0.53 in it (because the stub resolver is enabled, e.g. /etc/systemd/resolved.conf has a line like DNSStubListener=yes, or that line is commented out).

JeffPaine commented 4 months ago

This overall issue may have been complicated for several years by https://github.com/systemd/systemd/issues/14700 where setting DNSStubListener=no did not result in the symlink for /etc/resolv.conf being updated from /run/systemd/resolve/stub-resolv.conf to /run/systemd/resolve/resolv.conf until systemd v246 and later (as an example, Ubuntu LTS 20.04 doesn't include the fix, 22.04 does).

rsyring commented 4 months ago

Related to #7655?

luizkowalski commented 3 months ago

just to weigh in on this, I'm seeing exactly the same thing, but I think its...fine. I don't know, just double-check me:

I have a home server with Pi Hole running, which means I have to disable the DNS on my server and point to the router IP (because I want the server to also go through Pi Hole). This is done via ansible

- name: AA
  become: true
  lineinfile:
    dest: "/etc/systemd/resolved.conf"
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
    state: present
  with_items:
    - regexp: "^DNS="
      line: "DNS=192.168.178.1"
    - regexp: "^DNSStubListener"
      line: "DNSStubListener=no"
  notify:
    - restart systemd-resolved

The router, in turn, points the DNS to the server IP. All good here.

Tailscale overrides etc/resolv.conf

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search xxx fritz.box

Notice the "fritz.box", that's the router hostname

Systemd-resolved is just fine too:

● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-08-06 14:08:38 UTC; 3h 52min ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 526195 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 9259)
     Memory: 2.6M (peak: 2.9M)
        CPU: 306ms
     CGroup: /system.slice/systemd-resolved.service
             └─526195 /usr/lib/systemd/systemd-resolved

Aug 06 14:08:38 homeserver systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Aug 06 14:08:38 homeserver systemd-resolved[526195]: Positive Trust Anchors:
Aug 06 14:08:38 homeserver systemd-resolved[526195]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 06 14:08:38 homeserver systemd-resolved[526195]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.i>
Aug 06 14:08:38 homeserver systemd-resolved[526195]: Using system hostname 'homeserver'.
Aug 06 14:08:38 homeserver systemd[1]: Started systemd-resolved.service - Network Name Resolution

I can talk to other machines on the Tailnet, I can access internet too and I see the server as a client on Pi Hole

image
rufo commented 2 months ago

The current logic in manager_linux.go does work, however, when /etc/resolv.conf is symlinked to /run/systemd/resolve/stub-resolv.conf as the file should have a line like nameserver 127.0.0.53 in it (because the stub resolver is enabled, e.g. /etc/systemd/resolved.conf has a line like DNSStubListener=yes, or that line is commented out).

This worked for me. The file at /run/systemd/resolve/resolv.conf that was initially symlinked at /etc/resolv.conf had my local DNS servers, with Tailscale continually overwriting it and breaking DNS; changing the symlink to stub-resolv.conf and rebooting systemd-resolved and tailscaled kept everything intact. (This was on a Debian Bookworm system, so others' mileage may vary, of course.)

nitinmohan87 commented 1 month ago

In my case, my /etc/resolv.conf is symlinked as expected to /run/systemd/resolve/stub-resolv.conf. There is an entry to the local nameserver in /etc/resolv.conf - "nameserver 127.0.0.53".

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jan  3  2024 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search xxxx <------------ updated by tailscale. without tailscale running this line would say "search ."

I can confirm from journalctl log from tailscaled that it is using systemd-resolved

Oct 02 17:05:28 my-laptop tailscaled[1856]: dns: using "systemd-resolved" mode

I am still seeing my /etc/resolv.conf overwritten edited by tailscale. Is that expected? This is consequently affecting DNS resolution when creating Kind k8s cluster where access to external services from inside the kind cluster gets resolved to domains resolved by tailscale.

Versions Ubuntu 24.04 (standard installation) $ tailscale version 1.74.1 tailscale commit: ccd6bf2f4ee6421c23789b64e6e63c2ccbe87e08 other commit: 8fce4ce11275fcae33595c47076414129514e4ee go version: go1.23.1