DNS query over DBUS fails but stub works

[kadamski]

After moving to Fedora 33 which uses systemd-resolved I'm having problems with DNS on my local network if the DNS server is set to my local router (192.168.1.1). Once I force the DNS server to be 8.8.8.8, everything works correctly.

The resolving works if I do: $ dig kernel.org @192.168.1.1

it also works if I do: $ dig kernel.org @127.0.0.53

but it doesn't when I call: $ resolvectl query kernel.org

the process hangs for a long time with the last few syscalls being:

gettid()                                = 172053
futex(0x7f7c48ecbdb4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="DATA\r\nOK 4400f8ebbe151f3fd7f4a8d"..., iov_len=256}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 58
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1\0\0\0\0\1\0\0\0m\0\0\0\1\1o\0\25\0\0\0/org/fre"..., iov_len=128}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 128
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\2\1\1\r\0\0\0\377\377\377\377G\0\0\0\5\1u\0\1\0\0\0", iov_len=24}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 24
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\7\1s\0\24\0\0\0org.freedesktop.DBus\0\0\0\0"..., iov_len=77}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 77
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1 \0\0\0\2\0\0\0\242\0\0\0\1\1o\0\31\0\0\0/org/fre"..., iov_len=184}, {iov_base="\0\0\0\0\n\0\0\0kernel.org\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=32}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 216
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\4\1\1\r\0\0\0\377\377\377\377\227\0\0\0\7\1s\0\24\0\0\0", iov_len=24}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 24
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="org.freedesktop.DBus\0\0\0\0\6\1s\0\10\0\0\0"..., iov_len=157}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 157
recvmsg(3, {msg_namelen=0}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
ppoll([{fd=3, events=POLLIN}], 1, {tv_sec=119, tv_nsec=999711000}, NULL, 8

On the systemd-resolved side it looks like this:

Got message type=method_call sender=:1.65983 destination=org.freedesktop.resolve1 path=/org/freedesktop/resolve1 interface=org.freedesktop.resolve1.Manager member=ResolveHostname cookie=2 reply_cookie=0 signature=isit error-name=n/a error-message=n/a
idn2_lookup_u8: kernel.org → kernel.org
Looking up RR for kernel.org IN A.
Looking up RR for kernel.org IN AAAA.
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=4 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=GetNameOwner cookie=5 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.65982 path=n/a interface=n/a member=n/a cookie=4294967295 reply_cookie=5 signature=s error-name=n/a error-message=n/a
Switching to DNS server 192.168.1.1 for interface wlp3s0.
Cache miss for kernel.org IN AAAA
Transaction 14800 for <kernel.org IN AAAA> scope dns on wlp3s0/*.
Using feature level UDP+EDNS0 for transaction 14800.
Using DNS server 192.168.1.1 for transaction 14800.
Sending query packet with id 14800.
Cache miss for kernel.org IN A
Transaction 46054 for <kernel.org IN A> scope dns on wlp3s0/*.
Using feature level UDP+EDNS0 for transaction 46054.
Using DNS server 192.168.1.1 for transaction 46054.
Sending query packet with id 46054.
Got message type=method_return sender=org.freedesktop.DBus destination=:1.65982 path=n/a interface=n/a member=n/a cookie=4294967295 reply_cookie=4 signature= error-name=n/a error-message=n/a
Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',arg0=':1.65983' successfully installed.
Processing incoming packet on transaction 46054 (rcode=SUCCESS).
Verified we get a response at feature level UDP+EDNS0 from DNS server 192.168.1.1.
Added positive unauthenticated cache entry for kernel.org IN A 288s on wlp3s0/INET/192.168.1.1
Transaction 46054 for <kernel.org IN A> on scope dns on wlp3s0/* now complete with <success> from network (unsigned).
Timeout reached on transaction 14800.
Retrying transaction 14800.
Cache miss for kernel.org IN AAAA
Transaction 14800 for <kernel.org IN AAAA> scope dns on wlp3s0/*.
Using feature level UDP+EDNS0 for transaction 14800.
Sending query packet with id 14800.

$ resolvectl status gives me:

Global
       LLMNR setting: resolve             
MulticastDNS setting: no                  
  DNSOverTLS setting: no                  
      DNSSEC setting: no                  
    DNSSEC supported: no                  
Fallback DNS Servers: 1.1.1.1             
                      8.8.8.8             
                      1.0.0.1             
                      8.8.4.4             
                      2606:4700:4700::1111
                      2001:4860:4860::8888
                      2606:4700:4700::1001
                      2001:4860:4860::8844

Link 2 (enp0s25)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 3 (wlp3s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes                      
       LLMNR setting: yes                      
MulticastDNS setting: no                       
  DNSOverTLS setting: no                       
      DNSSEC setting: no                       
    DNSSEC supported: no                       
  Current DNS Server: 192.168.1.1              
         DNS Servers: 192.168.1.1              
          DNS Domain: ~.                       

Link 4 (virbr0)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Link 5 (virbr0-nic)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

What may be causing it? What else should I check to help resolving this issue?

[kadamski]

Sorry, wrong repo. I, obviously, should have reported that to main systemd repo.

[piotr-dobrogost]

@kadamski Please share a link to systemd issue after raising the problem there. Thanks.

[kadamski]

@piotr-dobrogost https://github.com/systemd/systemd/issues/18006