Slow DNS queries for service name w/o namespace

[amkartashov]

Nslookup is slow, query takes more than 10 seconds, query shows question section mismatch warnings

$ nslookup kong-kong-admin
;; ;; Question section mismatch: got kong-kong-admin.myns.svc.cluster.local/A/IN
Server:         172.19.80.1
Address:        172.19.80.1#53

Name:   kong-kong-admin.myns.svc.cluster.local
Address: 10.97.115.111
;; ;; Question section mismatch: got kong-kong-admin.myns.svc.cluster.local/AAAA/IN

To Reproduce

1. Connect to remote cluster
2. Create interceptor in ns myns
3. Run nslookup
4. It will be slow for 10 seconds, showing two warnings with Question section mismatch for A and AAAA queries.

Note that consequent request will work bc of cache. To reproduce the problem again wait for 1 minute so telepresence will do DNS cache cleanup.

Expected behavior

Fast queries, no warnings

Versions (please complete the following information):

• 2.11.1
• WSL2 with Ubuntu 22.04
• Remote single node cluster installed with kubeadm, 1.23

Additional context

Python services started with telepresence intercept fails when using async http requests, because of 5 seconds timeout: https://github.com/encode/httpx/discussions/2167 https://github.com/encode/httpx/discussions/2321

Workaround: set options timeout:1 attempts:2 in /etc/resolv.conf to force retry in 1 second - it will hit DNS cache in telepresence

REASON

Here https://github.com/telepresenceio/telepresence/blob/c495f52c451ea8efe69d7a371e6d6c7bf13d20b4/pkg/client/rootd/dns/server_linux.go#L95 you modify question section and it is returned back to the DNS client (dig/nslookup/application/etc). Since this is UDP (no sessions), response should have exact the same question section as query does to be matched, and response with other name in question section is rejected.

So what happens is:

• dns client->telepresence: give me A for kong-kong-admin (starts waiting)
• telepresence: this should be resolved in cluster, let's change name in question to kong-kong-admin.myns.svc.cluster.local and send it to cluster
• telepresence: got IP 10.97.115.111 from cluster
• telepresence->dns client: you asked me for kong-kong-admin.myns.svc.cluster.local, IP is 10.97.115.111
• dns client: hm, I was not asking for kong-kong-admin.myns.svc.cluster.local, let's discard this and show warning about question section mismatch
• dns client: waits for default timeout of 5 seconds
• dns client->telepresence: give me A for kong-kong-admin (retry)
• telepresence: oh, I have this in my cache
• telepresence->dns client: you asked me for kong-kong-admin, IP is 10.97.115.111
• dns client: I got A, let's check AAAA now ... situation is repeated, + another 5 seconds...

PATCH

The idea is simple: preserve original query and restore it back after DNS lookup in the cluster is finished.

--- a/pkg/client/rootd/dns/server_linux.go
+++ b/pkg/client/rootd/dns/server_linux.go
@@ -89,11 +89,14 @@ func (s *Server) resolveInSearch(c context.Context, q *dns.Question) (dnsproxy.R
        }

        if s.shouldApplySearch(query) {
+               origQuery := q.Name
                for _, sp := range s.search {
                        q.Name = query + sp
                        if rrs, rCode, err := s.resolveInCluster(c, q); err != nil || len(rrs) > 0 {
+                               q.Name = origQuery
                                return rrs, rCode, err
                        }
+                       q.Name = origQuery
                }
        }
        return s.resolveInCluster(c, q)

[amkartashov]

btw, I tested the patch and it works.

[amkartashov]

similar issue in coredns https://github.com/coredns/coredns/issues/1031

[thallgren]

@amkartashov great work, finding the problem and creating a fix for it. Thanks for doing that. I would recommend that you create a pull request with your fix. That way, you'll be listed as one of the contributors.

[amkartashov]

@amkartashov great work, finding the problem and creating a fix for it. Thanks for doing that. I would recommend that you create a pull request with your fix. That way, you'll be listed as one of the contributors.

done. Pls let me know if anything else is needed in PR. Not sure if this should be mentioned in changelog or testing fix on localhost can be considered as "adequately tested" in this case :)