kozyraki
commented
9 years ago I was able to recreate the behavior quite quickly after a restart of mesos-dns. Even in verbose mode I have no way of seeing what's wrong:
[christos@srv3.hw.ca1 mesos-dns]$ nslookup -port=31053 www.google.com 172.16.0.13 Server: 172.16.0.13 Address: 172.16.0.13#31053
Non-authoritative answer: Name: www.google.com Address: 64.233.171.147 Name: www.google.com Address: 64.233.171.99 Name: www.google.com Address: 64.233.171.103 Name: www.google.com Address: 64.233.171.104 Name: www.google.com Address: 64.233.171.105 Name: www.google.com Address: 64.233.171.106
[christos@srv3.hw.ca1 mesos-dns]$ nslookup -port=31053 chronos.marathon-0.7.6.mesos 172.16.0.13 ;; connection timed out; trying next origin ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached
Another worrisome issue is the statistics: VERBOSE: 2015/01/03 18:25:56 logging.go:32: {MesosRequests:3243 MesosSuccess:3243 MesosNXDomain:3242 MesosFailed:0 NonMesosRequests:477145 NonMesosSuccess:423833 NonMesosNXDomain:40987 NonMesosFailed:53312 NonMesosRecursed:112429}
This says that the vast majority of requests for the mesos domain return NXDomain. Even though the names used for the query show up in the verbose messages about records built. This is strange.
Given the way I have written the tests, MesosRequests and NonMesosRequests should also be equal (more or less at least). In this case they are off by a factor or 100x. Something is off here too.
After >24h of rather intense testing, I got mesos-dns to exhibit a erroneous behavior. Right now it is at the state where it serves external requests but not internal requests.
$ dig @172.16.0.13 -p 31053 chronos.marathon-0.7.6.mesos ; <<>> DiG 9.9.4-RedHat-9.9.4-14.el7_0.1 <<>> @172.16.0.13 -p 31053 chronos.marathon-0.7.6.mesos ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached
dig @172.16.0.13 -p 31053 www.kathimerini.gr
; <<>> DiG 9.9.4-RedHat-9.9.4-14.el7_0.1 <<>> @172.16.0.13 -p 31053 www.kathimerini.gr ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48876 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 7, ADDITIONAL: 11
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;www.kathimerini.gr. IN A
;; ANSWER SECTION: www.kathimerini.gr. 79524 IN A 88.198.40.186
;; AUTHORITY SECTION: gr. 78119 IN NS grdns-de.denic.de. gr. 78119 IN NS estia.ics.forth.gr. gr. 78119 IN NS gr-br.ics.forth.gr. gr. 78119 IN NS gr-at.ics.forth.gr. gr. 78119 IN NS gr-m.ics.forth.gr. gr. 78119 IN NS grdns.ics.forth.gr. gr. 78119 IN NS gr-ix.ics.forth.gr.
;; ADDITIONAL SECTION: gr-m.ics.forth.gr. 78119 IN A 194.0.4.10 gr-m.ics.forth.gr. 78119 IN AAAA 2001:678:7::4:10 estia.ics.forth.gr. 4274 IN A 139.91.191.3 estia.ics.forth.gr. 4274 IN AAAA 2001:648:2c30::191:3 gr-at.ics.forth.gr. 78119 IN A 78.104.145.227 gr-br.ics.forth.gr. 78119 IN A 200.160.7.163 gr-ix.ics.forth.gr. 78119 IN A 176.126.38.132 gr-ix.ics.forth.gr. 78119 IN AAAA 2001:7f8:6e:1::132 grdns.ics.forth.gr. 106442 IN A 139.91.1.1 grdns-de.denic.de. 78119 IN A 81.91.161.80
;; Query time: 3 msec ;; SERVER: 172.16.0.13#31053(172.16.0.13) ;; WHEN: Sat Jan 03 17:53:23 UTC 2015 ;; MSG SIZE rcvd: 690
There is absolutely nothing in stderror that can help me debug this (no messages about issues with internal requests). Let's add some.
We should also consider some periodic health messages on stderr or stdout. In addition to what we have right now, we should probably print # of internal records, print error messages when internal state is inconsistent, etc.
In general, scan through places where errors can happen and add some logging.