"!Backing off TLS on this upstream" on docker armv7, but not on docker amd64

[LuckyTurtleDev]

I hit the following error messages, if I run stubby in docker on my raspberry pi 2:

STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Conns=     0, Conn_fails=     1, Conn_shuts=      0, Backoffs     =     5
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Conns=     0, Conn_fails=     2, Conn_shuts=      0, Backoffs     =     5
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : !Backing off TLS on this upstream    - Will retry again in 2s at Wed Jun 30 14:09:42 2021
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : No valid upstreams for TLS... promoting this backed-off upstream for re-try...
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Conns=     0, Conn_fails=     1, Conn_shuts=      0, Backoffs     =     6
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Conns=     0, Conn_fails=     2, Conn_shuts=      0, Backoffs     =     6
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : !Backing off TLS on this upstream    - Will retry again in 2s at Wed Jun 30 14:09:42 2021
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : No valid upstreams for TLS... promoting this backed-off upstream for re-try...
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Conn closed: TLS - Resps=     0, Timeouts  =     0, Curr_auth =   None, Keepalive(ms)=     0
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Resps=     0, Timeouts  =     0, Best_auth =   None
STUBBY: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d      : Upstream   : TLS - Conns=     0, Conn_fails=     1, Conn_shuts=      0, Backoffs     =     7
STUBBY:    *FAILURE* no valid transports or upstreams available!
Could not schedule query: None of the configured upstreams could be used to send queries on the specified transports

But if I run the same image with the same config file on my tower pc everything works fine.

I did use the following docker-compose:

version: "3"

services:
  stubby:
    image: registry.gitlab.com/lukas1818/docker-stubby
    volumes:
      - ./stubby.yml:/etc/stubby/stubby.yml:ro
    ports:
      - 53:53/udp 
      - 53:53/tcp
    restart: unless-stopped

The dockerfile from the image:

FROM debian:stable-slim
LABEL git="https://gitlab.com/Lukas1818/docker-stubby"

RUN apt-get update && apt-get install -y \
    ca-certificates \
    stubby\
 && rm -rf /var/lib/apt/lists/*

COPY stubby.yml /etc/stubby/

EXPOSE 53/UDP

ENTRYPOINT ["/usr/bin/stubby","-l"]

https://gitlab.com/Lukas1818/docker-stubby/-/blob/master/Dockerfile

my stubbyfile:

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
edns_client_subnet_private : 1
round_robin_upstreams: 1
idle_timeout: 10000
appdata_dir: "/var/cache/stubby"
listen_addresses:
  - 0.0.0.0
upstream_recursive_servers:
  - address_data: 185.163.118.254
    tls_auth_name: "dns.lukas1818.de"
  - address_data: 2a03:4000:3b:1a:9838:ddff:feb6:ea5d
    tls_auth_name: "dns.lukas1818.de"

[saradickinson]

Can you use kdig or getdns_query from the raspberry pi to query your resolvers over TLS?

Can you also check your logs for lines like:

[10:44:29.022261] STUBBY: 145.100.185.15                           : Conn opened: TLS - Strict Profile
[10:44:29.132639] STUBBY: 145.100.185.15                           : Verify passed : TLS

This should Indicate if it is a local verification error. I'm also interested in to why you don't see the IPv4 addresses in the logs.....

[LuckyTurtleDev]

I did check the logs again and found the following lines at the start of the log:

STUBBY: 185.163.118.254                          : Conn opened: TLS - Strict Profile
STUBBY: 185.163.118.254                          : Verify failed : TLS - *Failure* -  (20) "unable to get local issuer certificate"
185.163.118.254                          : Conn closed: TLS - *Failure*

It seems I have simple overlook it last time. But I did not know, why there is a issue with the certificate. The container had installed ca-certificates and it does work on amd64.

I also found out, that I need to enable ipv6 explicit in docker. Sadly this is not possible for me, because docker need a manual set ipv6subnet, but my subnet does changes twice a day.

Can you use kdig or getdns_query from the raspberry pi to query your resolvers over TLS?

Should I run it inside the container or on the host? Sadly neither kdig nor getdns_query is in the package source of the host.

[saradickinson]

kdig is in the knot-dnsutils package. getdns_query is in the getdns package. FYI - Other docker images I have seen build stubby from source:

• https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Daemon+-+Stubby#DNSPrivacyDaemonStubby-DockerImages
• https://github.com/MatthewVance/stubby-docker

[LuckyTurtleDev]

I did run kdig inside the container and it does work without a issue:

# kdig +tls @dns.lukas1818.de lukas1818.de
;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 58492
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 4096 B; ext-rcode: NOERROR

;; QUESTION SECTION:
;; lukas1818.de.            IN  A

;; ANSWER SECTION:
lukas1818.de.           609 IN  A   185.163.118.254

;; Received 57 B
;; Time 2021-07-14 19:54:35 UTC
;; From 185.163.118.254@853(TCP) in 58.6 ms

I am a bit confused why stubby has a issue with the certificate, witch kdig does not hit.

Sorry it took me so long to reply.

[saradickinson]

They are build differently and use different TLS libraries, which pick up the cert store in different ways. The test shows that there are no other problems though, which is good.

I would try using on of the published docker files that build getdns from source so it links to openssl and the certs correctly e.g. https://github.com/MatthewVance/stubby-docker/blob/master/stubby/Dockerfile

[LuckyTurtleDev]

I notify that this issue does not happen, if I did build it on the raspberry itself without buildx.

So I think it is a buildx issue and not a stubby issue. Because of this I will close this issue now.

[pataquets]

@Lukas1818 : You might be interested in PR #310. Feedback on ARMv7 is much appreciated.

[LuckyTurtleDev]

@Lukas1818 : You might be interested in PR #310. Feedback on ARMv7 is much appreciated.

I have update my dockerfile to use alpine as base image. The issue does not happen there. Furthermore alpine is much smaller then debian, witch is used by the new official dockerfile.

Independent of this issue I plan to switch to trust-dns, witch does provide integrated cache.

[pataquets]

Glad to hear it worked, @Lukas1818. Too bad we'll not have the ARM feedback, OTOH. :( I wasn't aware of trust-dns, I'll check it out. Thanks for sharing!