Closed sandstrom closed 6 years ago
@sandstrom You could probably just disable the systemd-resolved DNS listeners like
[Resolve]
DNSStubListener=no
I think this should cause systemd-resolved to generate a resolv.conf like normal but not bind to 127.0.0.1:53 so that dnsmasq can.
dnsmasq should be either polling or using inotify to monitor changes to resolv.conf so any systemd-resolved updates show up with dnsmasq.
Let me know if that works for you and if so I can update the docs.
@mkeeler I tried that, but DNSStubListener=no
will only disable the listener itself, it won't change/update resolv.conf
(it will still contain a dns record for 127.0.0.53, which won't be listening).
There may be some additional command/configuration required to cause systemd-resolved to update the resolv.conf
file, that I'm not aware of. But DNSStubListener=no
plus service-reload won't do it.
Also, there is no binding conflict (that's not the issue), because the systemd-resolved stub will bind to 127.0.0.53
and dnsmasq will (by default) bind to 127.0.0.1
. The issue is that dns queries will never hit dnsmasq.
One ostensible solution would be to disable systemd-resolved's control over resolv.conf (while still having the stub running, bound to 127.0.0.53, and then update resolve.conf to point to dnsmasq, which in turn could be configured to fallback to 127.0.0.53 for namespaces it isn't configured for (non-consul queries).
However, I don't think that'll work. systemd-resolved accept dns queries via three interfaces. (1) a bus API, (2) a glibc api and (3) the stub listener on 127.0.0.53. So with the apparent solution above, one would still run the risk of having dns queries for services/applications handled by systemd (which is basically all of them) bypass the dnsmasq layer.
So instead we must find a way to configure systemd-resolved to query dnsmasq (or consul dns api directly) before any queries are forwarded up the default dns server chain (usually obtained via dhcp).
I hope this helps explain the issue in more detail!
Friendly ping @mkeeler 😄
@sandstrom Just getting back from a vacation and will look more into this soon. Disabling systemd-resolved in some cases where systemd-networkd is in use would break pushing down nameservers and domains via DHCP so its not a good solution. I have a hunch that something may be able to be done with nss configuration. As then all getaddrinfo/gethostbyname queries can be routed to the proper resolver. I still need to look into this some more especially as it relates to Ubuntu.
Another possible area to look into is network manager configuration (assuming you are using it). I think it has various functionality to use resolved or dnsmasq built into it.
@mkeeler I've been digging some more.
I agree that disabling systemd-resolved is a bad idea and should be avoided.
One avenue I've tried is to configure this only via systemd-network and its routing-only domain.
However, according to the top of the file networks can only be configured with one block, i.e. this would override other network settings (there doesn't seem to be a way to inject the routing-only domain setting into all networks, without disturbing the other, existing, network configuration).
# /etc/systemd/network/00-test.network
[Match]
# match everything
[Network]
DNS=127.0.0.54
Domains=~consul
Another idea is to put systemd-resolved into the fourth mode described in its man page, where other packages are responsible for /etc/resolv.conf
and systemd-resolved will only read and abide by it. In that case, we can simply set dnsmasq in that conf file and then configure dnsmasq to do forwarding similar to your current guide.
Thing I haven't figured out yet though, is how to let dnsmasq know about the upstream resolver (received via DHCP) that it should use for all queries that shouldn't be forwarded to consul. With NetworkManager there was an option for this, but Ubuntu 18.04 doesn't seem to be using NetworkManager any more.
Any thoughts on how to let dnsmasq know about the upstream server?
@sandstrom For dnsmasq to know about the upstreams you either have to manually put them in its configuration or have something add them to resolv.conf. It will (unless configured otherwise) watch for changes to resolv.conf and update its nameserver lists.
There are hooks for dhclient which could be used, for example something like this.
Basically one would then:
dhclient
such that it will update the config for dnsmasq (or bind) with new dns servers as they become available.Does that sound like a good flow?
Something you'd be willing to add instructions around to your guides on dns forwarding?
@mkeeler friendly ping 😄
@sandstrom That sounds like a decent flow. However I think I found an alternative.
In /etc/systemd/resolved.conf you can have:
DNS=127.0.0.1
Domains=~consul
Then have dnsmasq serving on 127.0.0.1:53 and doing its normal thing.
This should force systemd-resolved to send everything within the *.consul domain to the local system resolver and ignore resolvers configured via DHCP or via systemd-networkd per-link configurations.
I ran some tests with wireshark running and no DNS requests for .consul domains are ever sent out to any resolver other than 127.0.0.1:53.
To me this seems like the least invasive approach. What are your thoughts?
I did try to put this in resolved.conf but it doesn't parse
DNS=127.0.0.1:8600
Domains=~consul
Apparently systemd-resolved doesn't want port numbers in the DNS config. Looking at the systemd-resolved source code the specifically validate that the full string is an ipv4 or ipv6 address. Maybe if I find some time I might try to get a PR into systemd to fix that and then there would be no need for dnsmasq to run at all but instead we would have a direct systemd-resolved section of the configuration.
@mkeeler Yeah, I know (I tried something related, with non-53 ports). But you can have consul bind to port 53 on 127.0.0.54 and then config with DNS=127.0.0.54
.
Need to add this permission to your systemd unit file for consul to bind to low ports.
[Unit]
Description=Consul Agent
# …
[Service]
# …
AmbientCapabilities=CAP_NET_BIND_SERVICE
# …
[Install]
WantedBy=multi-user.target
To answer the other question: using your suggested resolved.conf
sounds good! As long as it works 😄 Haven't tested on our machine image yet, but will try! I'm happy with any solution that works reliably.
Yeah having systemd-resolved and dnsmasq involved seems a little bit of a hack when we can do it with just systemd-resolved.
I think the best course of action is to add a systemd-resolved section to that forwarding DNS doc and specify how to set it up with two options.
Instead of IP-tables I'd suggest using a systemd socket. I'll admit I haven't used it myself yet, it's the systemd method for handling low ports. So since we're in systemd land already (with systemd-resolved) it will make sense to use that mechanism.
Some more details:
@sandstrom I dont' think systemd socket activation is applicable here. You may be able to not use the "activation" part and still use systemd sockets with long running services but then it would require systemd specific modifications to Consul as well as modifications to the miekg/dns library we use to handle unix/file sockets and probably a few other things.
@mkeeler Yeah, I think you are right (read some more on sockets). Would require some modifications to Consul to accept sockets.
Although I think socket support would be useful, I don't think it should block this issue. IP-tables forwarding would certainly work.
@sandstrom I have a PR open with the documentation fixes if you wanted to take a peek. It probably wont hit the website until the end of June.
@mkeeler I'll have a look and also try this myself.
PTR records leaking out is just an effect of how reverse dns lookups work, right? (and those aren't part of "normal" consul usage)
@sandstrom Actually Consul will handle the .arpa domain for PTR record queries in addition to the configured consul domain. It will then use the configured recursors to resolve PTR records recursively if the IP is not known to Consul.
However with just systemd-resolved and Consul it isn't really possible to not expose all PTR queries to the main nameservers without manually configuring the recursors for Consul (which kind of defeats the purpose).
For the purposes of the guide it seemed a decent trade off since no consul specific information is being exposed to other nameservers. Just IP addresses.
@mkeeler I'll close this issue, the changes you PR:ed seems to solve this issue for us. Although we haven't rolled this into production just yet, I'm pretty sure it'll work (can reopen this issue in case it doesn't).
I still have my initial caveat on resolved.conf
, which I described in the top post:
DNS requests are sent to one of the listed DNS servers in parallel to suitable per-link DNS servers acquired from systemd-networkd.service(8) […] — https://www.freedesktop.org/software/systemd/man/resolved.conf.html
But I guess that description isn't valid when the DOMAINS=~consul
directive is also used (since you checked it with wireshark). Anyway, we're in need of a solution so we'll roll with this for now, and maybe revisit in the future in case something even better turns up.
Thanks for working with me on solving this! 💯
(I've added a few minor comments on your PR)
@sandstrom Yes assuming I am reading the docs right (and what I saw with wireshark is generally indicative of its normal behavior) using the ~
prefix should prevent the per-link configured DNS servers from being used.
After running with this setup for about a day I'm seeing this issue in the logs (below).
Seems like queries to resolve s3.eu-west-1.amazonaws.com is going through consul somehow, which I hadn't expected them to.
@mkeeler Any thoughts? Not 100% sure this is related to the changes above, but most likely. Perhaps we missed something in our configuration.
Jun 14 09:14:42 ubuntu-bionic consul[21810]: 2018/06/14 09:14:42 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 1 1} from client 127.0.0.1:36470 (udp)
Jun 14 09:14:43 ubuntu-bionic consul[21810]: 2018/06/14 09:14:43 [ERR] dns: recurse failed: dial udp 127.0.0.1:53: socket: too many open files
Jun 14 09:14:43 ubuntu-bionic consul[21810]: 2018/06/14 09:14:43 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 28 1} from client 127.0.0.1:43350 (udp)
Jun 14 09:14:43 ubuntu-bionic consul[21810]: 2018/06/14 09:14:43 [ERR] dns: recurse failed: dial udp 127.0.0.1:53: socket: too many open files
Jun 14 09:14:43 ubuntu-bionic consul[21810]: 2018/06/14 09:14:43 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 1 1} from client 127.0.0.1:60282 (udp)
Jun 14 09:14:44 ubuntu-bionic consul[21810]: 2018/06/14 09:14:44 [ERR] dns: recurse failed: dial udp 127.0.0.1:53: socket: too many open files
Jun 14 09:14:44 ubuntu-bionic consul[21810]: 2018/06/14 09:14:44 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 1 1} from client 127.0.0.1:44326 (udp)
Jun 14 09:14:44 ubuntu-bionic consul[21810]: 2018/06/14 09:14:44 [ERR] dns: recurse failed: dial udp 127.0.0.1:53: socket: too many open files
Jun 14 09:14:44 ubuntu-bionic consul[21810]: 2018/06/14 09:14:44 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 28 1} from client 127.0.0.1:40939 (udp)
Jun 14 09:14:46 ubuntu-bionic consul[21810]: 2018/06/14 09:14:46 [ERR] dns: recurse failed: dial udp 127.0.0.1:53: socket: too many open files
Jun 14 09:14:46 ubuntu-bionic consul[21810]: 2018/06/14 09:14:46 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 1 1} from client 127.0.0.1:53334 (udp)
Jun 14 09:14:46 ubuntu-bionic consul[21810]: 2018/06/14 09:14:46 [ERR] dns: recurse failed: dial udp 127.0.0.1:53: socket: too many open files
Jun 14 09:14:46 ubuntu-bionic consul[21810]: 2018/06/14 09:14:46 [ERR] dns: all resolvers failed for {s3.eu-west-1.amazonaws.com. 28 1} from client 127.0.0.1:47351 (udp)
vagrant@ubuntu-bionic:~$ sudo lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1063 root 3u IPv4 17656 0t0 TCP *:ssh (LISTEN)
sshd 1063 root 4u IPv6 17658 0t0 TCP *:ssh (LISTEN)
systemd-n 1635 systemd-network 22u IPv4 70689049 0t0 UDP ubuntu-bionic:bootpc
sshd 18507 root 3u IPv4 72929902 0t0 TCP ubuntu-bionic:ssh->_gateway:54235 (ESTABLISHED)
sshd 18622 vagrant 3u IPv4 72929902 0t0 TCP ubuntu-bionic:ssh->_gateway:54235 (ESTABLISHED)
node 20373 kibana 10u IPv4 71195 0t0 TCP localhost:36714->localhost:9200 (ESTABLISHED)
node 20373 kibana 11u IPv4 66949 0t0 TCP *:5601 (LISTEN)
node 20373 kibana 12u IPv4 71197 0t0 TCP localhost:36716->localhost:9200 (ESTABLISHED)
consul 21810 consul 3u IPv4 70662 0t0 TCP localhost:8300 (LISTEN)
consul 21810 consul 7u IPv4 70671 0t0 TCP localhost:8302 (LISTEN)
consul 21810 consul 8u IPv4 70672 0t0 UDP localhost:8302
consul 21810 consul 10u IPv4 70138 0t0 TCP localhost:8301 (LISTEN)
consul 21810 consul 11u IPv4 70139 0t0 UDP localhost:8301
consul 21810 consul 12u IPv4 70140 0t0 UDP localhost:8600
consul 21810 consul 13u IPv4 70675 0t0 TCP localhost:8600 (LISTEN)
consul 21810 consul 14u IPv4 70677 0t0 TCP localhost:8500 (LISTEN)
consul 21810 consul 15u IPv4 71064 0t0 TCP localhost:52419->localhost:8300 (ESTABLISHED)
consul 21810 consul 16u IPv4 71065 0t0 TCP localhost:8300->localhost:52419 (ESTABLISHED)
nginx 21878 root 6u IPv4 70158 0t0 TCP localhost:8099 (LISTEN)
nginx 21878 root 7u IPv6 70159 0t0 TCP ip6-localhost:8099 (LISTEN)
nginx 21878 root 8u IPv4 70160 0t0 TCP *:http-alt (LISTEN)
nginx 21878 root 9u IPv6 70161 0t0 TCP *:http-alt (LISTEN)
nginx 21879 www-data 6u IPv4 70158 0t0 TCP localhost:8099 (LISTEN)
nginx 21879 www-data 7u IPv6 70159 0t0 TCP ip6-localhost:8099 (LISTEN)
nginx 21879 www-data 8u IPv4 70160 0t0 TCP *:http-alt (LISTEN)
nginx 21879 www-data 9u IPv6 70161 0t0 TCP *:http-alt (LISTEN)
nginx 21880 www-data 6u IPv4 70158 0t0 TCP localhost:8099 (LISTEN)
nginx 21880 www-data 7u IPv6 70159 0t0 TCP ip6-localhost:8099 (LISTEN)
nginx 21880 www-data 8u IPv4 70160 0t0 TCP *:http-alt (LISTEN)
nginx 21880 www-data 9u IPv6 70161 0t0 TCP *:http-alt (LISTEN)
java 21975 elasticsearch 114u IPv6 71130 0t0 TCP ip6-localhost:9300 (LISTEN)
java 21975 elasticsearch 116u IPv6 71140 0t0 TCP localhost:9300 (LISTEN)
java 21975 elasticsearch 129u IPv6 71842 0t0 TCP ip6-localhost:9200 (LISTEN)
java 21975 elasticsearch 130u IPv6 71857 0t0 TCP localhost:9200->localhost:36714 (ESTABLISHED)
java 21975 elasticsearch 131u IPv6 71843 0t0 TCP localhost:9200 (LISTEN)
java 21975 elasticsearch 132u IPv6 71858 0t0 TCP localhost:9200->localhost:36716 (ESTABLISHED)
sshd 22548 root 3u IPv4 72924 0t0 TCP ubuntu-bionic:ssh->_gateway:55228 (ESTABLISHED)
sshd 22640 vagrant 3u IPv4 72924 0t0 TCP ubuntu-bionic:ssh->_gateway:55228 (ESTABLISHED)
systemd-r 28467 systemd-resolve 12u IPv4 905240 0t0 UDP localhost:domain
systemd-r 28467 systemd-resolve 13u IPv4 905241 0t0 TCP localhost:domain (LISTEN)
@mkeeler I think the issue may be that the configuration we've discussed will forward all lookups to consul, not only those under the .consul
tld (checked with nast
and dig consul.service.consul
plus dig google.com
, both will send packages to consul). This in conjunction with the consul recursors
setting (which we had set to 127.0.0.53) will cause a loop.
The thing is, we need recursors for some externally configured services, such as AWS RDS (hosted databases) and we don't want to stick a public dns in there (such as 8.8.8.8) since I don't think it will resolve AWS internal services correctly.
So we're sort of back at square one here 😄 (at least I am, perhaps you can see a way out of here?)
@sandstrom Given those error messages it looks like Consul is trying to use 127.0.0.1:53 as a recursor. This would be bad as 127.0.0.1:53 would be port forwarded back to itself creating an endless loop (and causing things to run out of sockets and have high CPU utilization).
I would think in the scenario with systemd-resolved consul shouldn't have any recursors configured as systemd-resolved should be handling all queries and then just forwarding some off to Consul.
Also, in this configuration Consul will be receiving all queries for this system. Without recursors set up it should just return a NXDOMAIN response and allow the other per-link resolvers to present a real answer.
@mkeeler Yes, exactly.
Dropping recursors
seems to work! 🎉 I didn't know it still worked with registered external services in the catalog without recursors.
Are there any downsides to having consul configured without recursors? (i.e. does it have any use-case besides being a fallback when all DNS queries are routed through consul)
@sandstrom recursors is just so Consul can forward queries for domains it doesn't know about to an upstream so you could use it for your primary DNS. So if you had a statically IPed system with a set of known name servers that don't change you could put the recursors into the Consul config and have the system only use 127.0.0.1:8600 for its primary DNS.
When fronting Consul with another DNS server there isn't much point to Consul also serving it out. Other downsides include increased network traffic and systemd load due to more concurrent queries being sent to upstream DNS servers. systemd-resolved will concurrently issue queries to both its upstream servers and Consul. Consul will then recurse and issue queries to more upstreams which should be unnecessary.
@sandstrom can't just add the VPC resolver for example? That should take care of RDS etc, if using Chef:
default['consul']['config']['recursors'] = [node['ipaddress'].split('.')[0..1].join('.').concat('.0.2')]
@scalp42 Yeah I know, we looked at that, it's a good idea! 😄 Works in the VPC, but wouldn't work on a vagrant machine (dev env). It also feels a bit hacky, would like to avoid it if we can (the idea with DHCP-provided DNS servers is a good one, we'd like to stick with it if we can).
In case anyone is stumbling upon this, I hit this issue in a local vagrant setup. I definitely had to setup a recursor to get systemd-resolved
to stop switching to itself for resolution - see this ticket for more details - but it's not clear what else I can do here other than generally complain about systemd 😅
This is what I did, for anyone that happens upon this. I think it's a better solution than what is presented above. The idea is to bind dnsmasq to a different IP address, and run dnsmasq and systemd-resolved in parallel, with systemd-resolved referring to dnsmasq as it's DNS server.
Note the DNS
address, below:
➜ cat /etc/systemd/resolved.conf
[Resolve]
DNS=127.0.0.2
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=yes
#DNSStubListener=yes
And we change the listen address for dnsmasq.conf
:
➜ cat /etc/dnsmasq.conf | grep 127.0.0.2
listen-address=127.0.0.2
And you need to tell dnsmasq to ignore systemd-resolved to prevent a loop:
➜ cat /etc/default/dnsmasq
# This file has five functions:
# 1) to completely disable starting dnsmasq,
# 2) to set DOMAIN_SUFFIX by running `dnsdomainname`
# 3) to select an alternative config file
# by setting DNSMASQ_OPTS to --conf-file=<file>
# 4) to tell dnsmasq to read the files in /etc/dnsmasq.d for
# more configuration variables.
# 5) to stop the resolvconf package from controlling dnsmasq's
# idea of which upstream nameservers to use.
# For upgraders from very old versions, all the shell variables set
# here in previous versions are still honored by the init script
# so if you just keep your old version of this file nothing will break.
DOMAIN_SUFFIX=``
#DNSMASQ_OPTS="--conf-file=/etc/dnsmasq.alt"
# Whether or not to run the dnsmasq daemon; set to 0 to disable.
ENABLED=1
# By default search this drop directory for configuration options.
# Libvirt leaves a file here to make the system dnsmasq play nice.
# Comment out this line if you don't want this. The dpkg-* are file
# endings which cause dnsmasq to skip that file. This avoids pulling
# in backups made by dpkg.
CONFIG_DIR=/etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
# If the resolvconf package is installed, dnsmasq will use its output
# rather than the contents of /etc/resolv.conf to find upstream
# nameservers. Uncommenting this line inhibits this behaviour.
# Note that including a "resolv-file=<filename>" line in
# /etc/dnsmasq.conf is not enough to override resolvconf if it is
# installed: the line below must be uncommented.
IGNORE_RESOLVCONF=yes
Thanks so much @SwitchedToGitlab
To anybody here because they want to integrate with consul, add the following:
# dnsmasq.conf
server=/consul$/127.0.0.1#8600
@SwitchedToGitlab Thanks for sharing your approach. Worked for me, and solved reverse lookup issue when queries round robin between consul and upstream DNS.
One thing to add, I had to uncomment bind-interfaces
in /etc/dnsmasq.conf
Thanks for all this. On my system, which was built for AWS using packer and terraform module consul-cluster
, I found that the only reason I need to move dnsmasq
to 127.0.0.2 is because something (run-consul
?) is setting up iptables
rules that forward 127.0.0.1:53 to 127.0.0.1:8600.
As Shaggy said, "it wasn't me."
Which all makes sense with only systemd-resolved
and consul
in the picture, but makes much less sense with dnsmasq
included.
root@ip-10-0-0-26:~# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT udp -- anywhere localhost udp dpt:domain redir ports 8600
REDIRECT tcp -- anywhere localhost tcp dpt:domain redir ports 8600
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
(UPDATE) NAT setup is here
Which all makes sense with only
systemd-resolved
andconsul
in the picture, but makes much less sense withdnsmasq
included.
@tpdownes do you have a use case that warrants installing both dnsmasq
and systemd-resolved
over just using this: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved?
https://github.com/hashicorp/consul/issues/6462 describes one here.
Not my use case, but it appears to be the only way to ensure reverse lookup never goes through consul.
FWIW - i would describe that as a systemd-resolved
problem (you configured a DNS server to perform forward resolution for .consul
but it sneaks in and does reverse lookup, too). Nevertheless, that's the only reason I can see for having dnsmasq
installed. You might also read my PR https://github.com/hashicorp/consul/pull/6731 on improving reverse lookup when you want only consul to perform reverse lookup on the subnets it's a part of.
@sarkis there's a use case for having /etc/hosts.consul
be read by dnsmasq
to lower the load on the Consul resolver (without getting into too many details).
Just a FYI, it's possible to use systed.resolvd
w/ consul-agent. See here:
https://gist.github.com/kquinsland/5cdc63614a581d9b392f435740b58729
Hey there,
This issue has been automatically locked because it is closed and there hasn't been any activity for at least 30 days.
If you are still experiencing problems, or still have questions, feel free to open a new one :+1:.
Overview of the Issue
The consul guides explain how to run dnsmasq as root, forwarding certain requests to consul. This worked well before systemd (e.g. ubuntu 14.04 and 16.04), but with more recent linux distros running systemd (ubuntu 17.04 and 18.04 LTS) the current guide won't work.
systemd-resolved will control
/etc/resolv.conf
(it's just a symlink now), so dig will use systemd-resolved by default, instead of dnsmasq.I see two solutions here:
All three of them would require guide updates. A related issue is this one (https://github.com/hashicorp/consul/issues/3945).
One partial workaround is to configure the following:
In that case resolved will ask dnsmasq (indirectly consul) simultaneously with asking its upstream dns (usually from dhcp), which seems to work. However, that would leak consul dns queries outside the local machine. It will probably also be brittle, i.e. it relies on the response from dnsmasq coming in before those from the per-link dns servers.
This leaking feels like a rather big downside.
Reproduction Steps