jwardle
commented
5 years ago Further to the above I have found that one of the nameservers in /etc/resolv.conf within the containers does not expose a DNS server, the second nameserver does (it's the service fabric host node, with the DNSService running). When altering the resolv.conf to remove the faulty nameserver results in no requests being successful, not even intermittently. Setting the Kong resolver IP to the 'correct' nameserver IP via KONG_DNS_RESOLVER Env var also has the same effect. I find this strange, as when I remove the faulty nameserver from resolve.conf Dig will return correct results, nslookup continues to function - the only thing that cannot resovle the URL is Kong! With the faulty nameserver left in the resolve.conf file then Dig will return SERVFAIL (as it hits the first nameserver which does not actually expose the DNS service I assume).
/etc/resolv.conf
nameserver 10.14.2.101 <—- Working DNS server able to resolve upstream.hostname
nameserver 172.20.48.1 <--- Faulty IP without a DNS server exposed. Have removed/reordered resolv.conf however no luck
search searchdomain.localSuccessful Dig results for the upstream domain
/ # dig upstream.hostname
; <<>> DiG 9.11.6-P1 <<>> upstream.hostname
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50344
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 8192
;; QUESTION SECTION:
;upstream.hostname. IN A
;; ANSWER SECTION:
upstream.hostname. 1 IN A 10.14.2.101
upstream.hostname. 1 IN A 10.14.2.100
;; Query time: 3 msec
;; SERVER: 10.14.2.101#53(10.14.2.101)
;; WHEN: Sun Jun 23 16:41:09 UTC 2019
;; MSG SIZE rcvd: 154I'm really struggling to understand why Dig/nslookup is different from what Kong is seeing. Is there a way to log the DNS resolve requests/responses, or at least the IP it is challenging?
Tieske
psrank
Summary
Kong 1.2.0 intermittently fails to resolve the upstream service's hostname, and therefore fails to process the request resulting in either an
An unexpected error occurredorname resolution failuremessage being returned to the client. The resolution appears to happen for 5-10 seconds after which requests are serviced as expected for ~30 seconds, and then it fails again. This cycle is continuous and I do not understand what is driving it yet.The kong Docker image 1.2.0-alpine is being used, deployed within a Standalone Azure Service Fabric cluster of 3 nodes. The DnsService is running within the cluster, and when queried from the localhost or within the Kong container the upstream service's hostname is successfully resolved. i.e. calling
nslookup upstream.hostalways successfully retrieves, and the same for acurl upstream.host:8100. Theresolve.confof the Kong container has the correct nameservers inherited from Service Fabric - as expected given the nslookup etc. succeeds every time.I am using a
kong.yml(below) declarative configuration in DB-less mode with a basic configuration.The error I am repeatedly seeing is (also in the logs below):
Steps To Reproduce
kong.yml(in Service Fabric, however don't believe this is the cause currently as DNS settings appear to be working consistently whilst this issue is being experienced from a Kong perspective)Additional Details & Logs
1.2.0)1.2.0-alpine), running on Azure SF Windows 2019Note: The below configuration & logs has been sanitised for sensistive info.
Kong YML
Kong Configuration from Admin endpoint
Kong Logs
Note: When you see a 404 response in the below logs this is a successful response from the upstream backend server being returned. Disregard the fact it's actually a 404 error.