Listening conflict between systemd-resolved and pdns since systemd 231

[HLFH]

Hi,

With pdns 4.0.1 on ArchLinux.

pdns_control does not answer at all or rarely.

[root@nssec ~]# pdns_control current-config
Timeout error: Error from remote in receive(): Resource temporarily unavailable
[root@nssec ~]# pdns_control help
Timeout error: Error from remote in receive(): Resource temporarily unavailable
[root@nssec ~]# 

My almost working systemd file:

[Unit]
Description=PowerDNS Authoritative Server
Documentation=man:pdns_server(1) man:pdns_control(1)
Documentation=https://doc.powerdns.com
Wants=network-online.target
After=network-online.target mysqld.service

[Service]
Type=forking
ExecStart=/usr/bin/pdns_server --guardian=yes --daemon=yes --disable-syslog --write-pid=yes
Restart=on-failure
RestartSec=1
StartLimitInterval=0
PrivateTmp=true
PrivateDevices=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_CHOWN CAP_SYS_CHROOT
NoNewPrivileges=true
# ProtectSystem=full will disallow write access to /etc and /usr, possibly
# not being able to write slaved-zones into sqlite3 or zonefiles.
ProtectSystem=full
ProtectHome=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

[Install]
WantedBy=multi-user.target

The default broken systemd file I don't use because of https://github.com/PowerDNS/pdns/issues/4180 and https://github.com/PowerDNS/pdns/issues/2307:

[Unit]
Description=PowerDNS Authoritative Server
Documentation=man:pdns_server(1) man:pdns_control(1)
Documentation=https://doc.powerdns.com
Wants=network-online.target
After=network-online.target mysqld.service postgresql.service slapd.service

[Service]
Type=simple
ExecStart=/usr/bin/pdns_server --guardian=no --daemon=no --disable-syslog --write-pid=no
Restart=on-failure
RestartSec=1
StartLimitInterval=0
PrivateTmp=true
PrivateDevices=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_CHOWN CAP_SYS_CHROOT
NoNewPrivileges=true
# ProtectSystem=full will disallow write access to /etc and /usr, possibly
# not being able to write slaved-zones into sqlite3 or zonefiles.
ProtectSystem=full
ProtectHome=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

[Install]
WantedBy=multi-user.target

My anonymised /etc/powerdns/pdns.conf:

# Autogenerated configuration file template
#################################
# allow-axfr-ips    Allow zonetransfers only to these subnets
#
allow-axfr-ips=$MYIPV6,$MYIPV4

#################################
# allow-dnsupdate-from  A global setting to allow DNS updates from these IP ranges.
#
# allow-dnsupdate-from=127.0.0.0/8,::1

#################################
# allow-notify-from Allow AXFR NOTIFY from these IP ranges. If empty, drop all incoming notifies.
#
# allow-notify-from=0.0.0.0/0,::/0

#################################
# allow-recursion   List of subnets that are allowed to recurse
#
# allow-recursion=

#################################
# allow-unsigned-notify Allow unsigned notifications for TSIG secured domains
#
# allow-unsigned-notify=yes

#################################
# allow-unsigned-supermaster    Allow supermasters to create zones without TSIG signed NOTIFY
#
# allow-unsigned-supermaster=yes

#################################
# also-notify   When notifying a domain, also notify these nameservers
#
# also-notify=

#################################
# any-to-tcp    Answer ANY queries with tc=1, shunting to TCP
#
# any-to-tcp=no

#################################
# api   Enable/disable the REST API
#
api=yes

#################################
# api-key   Static pre-shared authentication key for access to the REST API
#
api-key=mypassword

#################################
# api-logfile   Location of the server logfile (used by the REST API)
#
# api-logfile=/var/log/pdns.log

#################################
# api-readonly  Disallow data modification through the REST API when set
#
# api-readonly=no

#################################
# cache-ttl Seconds to store packets in the PacketCache
#
cache-ttl=6

#################################
# carbon-interval   Number of seconds between carbon (graphite) updates
#
# carbon-interval=30

#################################
# carbon-ourname    If set, overrides our reported hostname for carbon stats
#
# carbon-ourname=

#################################
# carbon-server If set, send metrics in carbon (graphite) format to this server
#
# carbon-server=

#################################
# chroot    If set, chroot to this directory for more security
#
# chroot=

#################################
# config-dir    Location of configuration directory (pdns.conf)
#
config-dir=/etc/powerdns

#################################
# config-name   Name of this virtual configuration - will rename the binary image
#
# config-name=

#################################
# control-console   Debugging switch - don't use
#
# control-console=no

#################################
# daemon    Operate as a daemon
#
# daemon=no

#################################
# default-ksk-algorithms    Default KSK algorithms
#
# default-ksk-algorithms=

#################################
# default-ksk-size  Default KSK size (0 means default)
#
# default-ksk-size=0

#################################
# default-soa-edit  Default SOA-EDIT value
#
default-soa-edit=INCEPTION-INCREMENT

#################################
# default-soa-edit-signed   Default SOA-EDIT value for signed zones
#
default-soa-edit-signed=INCEPTION-INCREMENT

#################################
# default-soa-mail  mail address to insert in the SOA record if none set in the backend
#
default-soa-mail=hostmaster.example.com

#################################
# default-soa-name  name to insert in the SOA record if none set in the backend
#
default-soa-name=nssec.example.com

#################################
# default-ttl   Seconds a result is valid if not set otherwise
#
default-ttl=86400

#################################
# default-zsk-algorithms    Default ZSK algorithms
#
default-zsk-algorithms=rsasha512

#################################
# default-zsk-size  Default ZSK size (0 means default)
#
default-zsk-size=1025

#################################
# direct-dnskey Fetch DNSKEY RRs from backend during DNSKEY synthesis
#
# direct-dnskey=no

#################################
# disable-axfr  Disable zonetransfers but do allow TCP queries
#
disable-axfr=no

#################################
# disable-axfr-rectify  Disable the rectify step during an outgoing AXFR. Only required for regression testing.
#
# disable-axfr-rectify=no

#################################
# disable-tcp   Do not listen to TCP queries
#
# disable-tcp=no

#################################
# distributor-threads   Default number of Distributor (backend) threads to start
#
distributor-threads=3

#################################
# dname-processing  If we should support DNAME records
#
# dname-processing=no

#################################
# dnsupdate Enable/Disable DNS update (RFC2136) support. Default is no.
#
# dnsupdate=no

#################################
# do-ipv6-additional-processing Do AAAA additional processing
#
do-ipv6-additional-processing=yes

#################################
# edns-subnet-processing    If we should act on EDNS Subnet options
#
# edns-subnet-processing=no

#################################
# entropy-source    If set, read entropy from this file
#
# entropy-source=/dev/urandom

#################################
# experimental-lua-policy-script    Lua script for the policy engine
#
# experimental-lua-policy-script=

#################################
# forward-dnsupdate A global setting to allow DNS update packages that are for a Slave domain, to be forwarded to the master.
#
# forward-dnsupdate=yes

#################################
# guardian  Run within a guardian process
#
# guardian=no

#################################
# include-dir   Include *.conf files from this directory
#
# include-dir=

#################################
# launch    Which backends to launch and order to query them in
#
launch=gmysql
gmysql-dnssec=yes
gmysql-host=localhost
gmysql-socket=/run/mysqld/mysqld.sock
gmysql-user=myuser
gmysql-password=mypassword
gmysql-dbname=pdns

#################################
# load-modules  Load this module - supply absolute or relative path
#
# load-modules=

#################################
# local-address Local IP addresses to which we bind
#
local-address=0.0.0.0

#################################
# local-address-nonexist-fail   Fail to start if one or more of the local-address's do not exist on this server
#
# local-address-nonexist-fail=yes

#################################
# local-ipv6    Local IP address to which we bind
#
local-ipv6=::

#################################
# local-ipv6-nonexist-fail  Fail to start if one or more of the local-ipv6 addresses do not exist on this server
#
# local-ipv6-nonexist-fail=yes

#################################
# local-port    The port on which we listen
#
local-port=53

#################################
# log-dns-details   If PDNS should log DNS non-erroneous details
#
log-dns-details=yes

#################################
# log-dns-queries   If PDNS should log all incoming DNS queries
#
log-dns-queries=yes

#################################
# logging-facility  Log under a specific facility
#
# logging-facility=

#################################
# loglevel  Amount of logging. Higher is more. Do not set below 3
#
loglevel=10

#################################
# lua-prequery-script   Lua script with prequery handler (DO NOT USE)
#
# lua-prequery-script=

#################################
# master    Act as a master
#
master=yes

#################################
# max-cache-entries Maximum number of cache entries
#
# max-cache-entries=1000000

#################################
# max-ent-entries   Maximum number of empty non-terminals in a zone
#
# max-ent-entries=100000

#################################
# max-nsec3-iterations  Limit the number of NSEC3 hash iterations
#
# max-nsec3-iterations=500

#################################
# max-queue-length  Maximum queuelength before considering situation lost
#
# max-queue-length=5000

#################################
# max-signature-cache-entries   Maximum number of signatures cache entries
#
# max-signature-cache-entries=

#################################
# max-tcp-connections   Maximum number of TCP connections
#
# max-tcp-connections=10

#################################
# module-dir    Default directory for modules
#
module-dir=/usr/lib/powerdns

#################################
# negquery-cache-ttl    Seconds to store negative query results in the QueryCache
#
negquery-cache-ttl=60

#################################
# no-shuffle    Set this to prevent random shuffling of answers - for regression testing
#
# no-shuffle=off

#################################
# non-local-bind    Enable binding to non-local addresses by using FREEBIND / BINDANY socket options
#
# non-local-bind=no

#################################
# only-notify   Only send AXFR NOTIFY to these IP addresses or netmasks
#
# only-notify=0.0.0.0/0,::/0

#################################
# out-of-zone-additional-processing Do out of zone additional processing
#
# out-of-zone-additional-processing=yes

#################################
# overload-queue-length Maximum queuelength moving to packetcache only
#
# overload-queue-length=0

#################################
# prevent-self-notification Don't send notifications to what we think is ourself
#
# prevent-self-notification=yes

#################################
# query-cache-ttl   Seconds to store query results in the QueryCache
#
# query-cache-ttl=20

#################################
# query-local-address   Source IP address for sending queries
#
# query-local-address=0.0.0.0

#################################
# query-local-address6  Source IPv6 address for sending queries
#
# query-local-address6=::

#################################
# query-logging Hint backends that queries should be logged
#
# query-logging=no

#################################
# queue-limit   Maximum number of milliseconds to queue a query
#
# queue-limit=1500

#################################
# receiver-threads  Default number of receiver threads to start
#
# receiver-threads=1

#################################
# recursive-cache-ttl   Seconds to store packets for recursive queries in the PacketCache
#
# recursive-cache-ttl=10

#################################
# recursor  If recursion is desired, IP address of a recursing nameserver
#
# recursor=

#################################
# retrieval-threads Number of AXFR-retrieval threads for slave operation
#
# retrieval-threads=2

#################################
# reuseport Enable higher performance on compliant kernels by using SO_REUSEPORT allowing each receiver thread to open its own socket
#
# reuseport=no

#################################
# security-poll-suffix  Domain name from which to query security update notifications
#
# security-poll-suffix=secpoll.powerdns.com.

#################################
# send-root-referral    Send out old-fashioned root-referral instead of ServFail in case of no authority
#
# send-root-referral=no

#################################
# server-id Returned when queried for 'server.id' TXT or NSID, defaults to hostname - disabled or custom
#
server-id=localhost

#################################
# setgid    If set, change group id to this gid for more security
#
setgid=nobody

#################################
# setuid    If set, change user id to this uid for more security
#
setuid=nobody

#################################
# signing-threads   Default number of signer threads to start
#
# signing-threads=3

#################################
# slave Act as a slave
#
slave=no

#################################
# slave-cycle-interval  Schedule slave freshness checks once every .. seconds
#
# slave-cycle-interval=60

#################################
# slave-renotify    If we should send out notifications for slaved updates
#
# slave-renotify=no

#################################
# soa-expire-default    Default SOA expire
#
soa-expire-default=2419200

#################################
# soa-minimum-ttl   Default SOA minimum ttl
#
soa-minimum-ttl=10800

#################################
# soa-refresh-default   Default SOA refresh
#
soa-refresh-default=28800

#################################
# soa-retry-default Default SOA retry
#
soa-retry-default=7200

#################################
# socket-dir    Where the controlsocket will live, /var/run when unset and not chrooted
#
socket-dir=/var/run

#################################
# tcp-control-address   If set, PowerDNS can be controlled over TCP on this address
#
# tcp-control-address=

#################################
# tcp-control-port  If set, PowerDNS can be controlled over TCP on this address
#
# tcp-control-port=53000

#################################
# tcp-control-range If set, remote control of PowerDNS is possible over these networks only
#
# tcp-control-range=127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fe80::/10

#################################
# tcp-control-secret    If set, PowerDNS can be controlled over TCP after passing this secret
#
# tcp-control-secret=

#################################
# traceback-handler Enable the traceback handler (Linux only)
#
# traceback-handler=yes

#################################
# trusted-notification-proxy    IP address of incoming notification proxy
#
# trusted-notification-proxy=

#################################
# udp-truncation-threshold  Maximum UDP response size before we truncate
#
# udp-truncation-threshold=1680

#################################
# version-string    PowerDNS version in packets - full, anonymous, powerdns or custom
#
# version-string=full

#################################
# webserver Start a webserver for monitoring
#
webserver=yes

#################################
# webserver-address IP Address of webserver to listen on
#
webserver-address=::

#################################
# webserver-allow-from  Webserver access is only allowed from these subnets
#
webserver-allow-from=0.0.0.0/0,::/0

#################################
# webserver-password    Password required for accessing the webserver
#
# webserver-password=

#################################
# webserver-port    Port of webserver to listen on
#
webserver-port=8081

#################################
# webserver-print-arguments If the webserver should print arguments
#
# webserver-print-arguments=no

#################################
# write-pid Write a PID file
# write-pid=yes

How to fix this issue ("Timeout error: Error from remote in receive(): Resource temporarily unavailable")?

Thanks, HLFH

[bjoe2k4]

I am using the default pdns.service file as shipped with arch with one modification: Change back to Type=notify (which is the pdns default) and remove (see also #4179 and #4161) the chroot= setting which is located at the bottom of pdns.conf and originates from packaging. Everything works out just fine for me.

[HLFH]

@bjoe2k4 Are you talking exactly about this systemd file?

[Unit]
Description=PowerDNS Authoritative Server
Documentation=man:pdns_server(1) man:pdns_control(1)
Documentation=https://doc.powerdns.com
Wants=network-online.target
After=network-online.target mysqld.service postgresql.service slapd.service

[Service]
Type=notify
ExecStart=/usr/bin/pdns_server --guardian=no --daemon=no --disable-syslog --write-pid=no
Restart=on-failure
RestartSec=1
StartLimitInterval=0
PrivateTmp=true
PrivateDevices=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_CHOWN CAP_SYS_CHROOT
NoNewPrivileges=true
# ProtectSystem=full will disallow write access to /etc and /usr, possibly
# not being able to write slaved-zones into sqlite3 or zonefiles.
ProtectSystem=full
ProtectHome=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

[Install]
WantedBy=multi-user.target

But by using this systemd file, I'm getting this issue:

-- L'unité (unit) pdns.service a commencé à démarrer.
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 Reading random entropy from '/dev/urandom'
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 Loading '/usr/lib/powerdns/libgmysqlbackend.so'
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 [gmysqlbackend] This is the gmysql backend version 4.0.1 reporting
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 This is a standalone pdns
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 Listening on controlsocket in '/var/run/pdns.controlsocket'
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 Unable to bind UDP socket to '0.0.0.0:53': Address already in use
sept. 25 14:35:41 nssec.hlfh.space pdns_server[848]: Sep 25 14:35:41 Fatal error: Unable to bind to UDP socket
sept. 25 14:35:41 nssec.hlfh.space systemd[1]: pdns.service: Main process exited, code=exited, status=1/FAILURE
sept. 25 14:35:41 nssec.hlfh.space systemd[1]: Failed to start PowerDNS Authoritative Server.
-- Subject: L'unité (unit) pdns.service a échoué
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- L'unité (unit) pdns.service a échoué, avec le résultat failed.
sept. 25 14:35:41 nssec.hlfh.space systemd[1]: pdns.service: Unit entered failed state.
sept. 25 14:35:41 nssec.hlfh.space systemd[1]: pdns.service: Failed with result 'exit-code'.

So, I'm getting then the issue: "Unable to bind UDP socket to '0.0.0.0:53': Address already in use [...] Fatal error: Unable to bind to UDP socket"

[bjoe2k4]

Whats systemctl status systemd-resolved.service ?

[HLFH]

Now it's working THANKS if I'm stopping systemd-resolved.service. And I don't understand why. Couldn't we use pdnsAND systemd-resolved?

[bjoe2k4]

You can, but you have to bind pdns to your external interface. systemd-resolved is listening on 127.0.0.53:53. Therefore you cannot listen on 0.0.0.0:53 as this also includes 127.0.0.53:53.

[HLFH]

Seems related to https://github.com/systemd/systemd/pull/4061, @poettering again :)

[bjoe2k4]

I'd rather say @usererror ;-) The same would have happened if you'd install any local resolver like pdns-recursor or dnsmasq.

[HLFH]

But pdns-recursor is a recursive DNS resolver, and systemd-resolved is only a stub DNS resolver. Furthermore, this issue is only happening since systemd 231 released on 2016-07-25:

systemd-resolved now listens on the local IP address 127.0.0.53:53
for DNS requests. This improves compatibility with local programs
that do not use the libc NSS or systemd-resolved's bus APIs for name
resolution. This minimal DNS service is only available to local
programs and does not implement the full DNS protocol, but enough to
cover local DNS clients. A new, static resolv.conf file, listing just
this DNS server is now shipped in /usr/lib/systemd/resolv.conf. It is
now recommended to make /etc/resolv.conf a symlink to this file in
order to route all DNS lookups to systemd-resolved, regardless if
done via NSS, the bus API or raw DNS packets. Note that this local
DNS service is not as fully featured as the libc NSS or
systemd-resolved's bus APIs. For example, as unicast DNS cannot be
used to deliver link-local address information (as this implies
sending a local interface index along), LLMNR/mDNS support via this
interface is severely restricted. It is thus strongly recommended for
all applications to use the libc NSS API or native systemd-resolved
bus API instead.

As you can see here.

In fact, this change has created a conflict with authoritative DNS servers like PowerDNS and we need this issue solved. So YES, it's @poettering. And I'm just saying that systemd has done an interesting change...that needs to be improved and fixed to avoid any conflict between the stub resolver systemd-resolved and the authoritative resolver pdns.

[Habbie]

@HLFH want to change the ticket title to cover the actual issue here?

[HLFH]

The workaround is thanks to @bjoe2k4:

systemctl stop systemd-resolved
systemctl disable systemd-resolved

vim /etc/systemd/system/pdns.service
[Unit]
Description=PowerDNS Authoritative Server
Documentation=man:pdns_server(1) man:pdns_control(1)
Documentation=https://doc.powerdns.com
Wants=network-online.target
After=network-online.target mysqld.service postgresql.service slapd.service

[Service]
Type=notify
ExecStart=/usr/bin/pdns_server --guardian=no --daemon=no --disable-syslog --write-pid=no
Restart=on-failure
RestartSec=1
StartLimitInterval=0
PrivateTmp=true
PrivateDevices=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_CHOWN CAP_SYS_CHROOT
NoNewPrivileges=true
# ProtectSystem=full will disallow write access to /etc and /usr, possibly
# not being able to write slaved-zones into sqlite3 or zonefiles.
ProtectSystem=full
ProtectHome=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

[Install]
WantedBy=multi-user.target
:wq!

systemctl daemon-reload
systemctl restart pdns

@Habbie Done. I'm keeping this issue open until the merge of https://github.com/systemd/systemd/pull/4061 into systemd.

[HLFH]

Hum, it will be released in systemd 232.