Exploring DNS queries

[arlake228]

Original issue 407 created by arlake228 on 2010-02-25T15:20:19.000Z:

Opening an exploratory bug on the glut of DNS queries seen by OU/IU in USATLAS. I am not sure this is a defect as of yet, so lets do some investigating. Story so far:

OU and IU both reported seeing machines that were tagged as making lots of DNS queries. IU was pre-3.1.2 and OU appears to be both 3.1.1 and 3.1.2. Here are some numbers from OU. For legend purposes:

ps1.ochep.ou.edu = 129.15.40.231 = Latency ps2.ochep.ou.edu = 129.15.40.232 = Throughput

The results are for a 48 hour period, and split between OUs 4 DNS servers. The '-rev' tag indicates reverse DNS on itself:

129.15.1.20: 129.15.40.231 - 4516611 total, throughout 129.15.40.232 - NONE 129.15.40.231-rev - 1052883 total, 1052875 from itself 129.15.40.232-rev - 11 total, most from 129.15.40.231

129.15.1.10: 129.15.40.231 - NONE 129.15.40.232 - NONE 129.15.40.231-rev - outsiders, few 129.15.40.232-rev - outsiders, few

129.15.1.21: 129.15.40.231 - 1377 total, throughout 129.15.40.232 - NONE 129.15.40.231-rev - 207, all but 5 from 231 129.15.40.232-rev - outsiders, few

129.15.1.9: 129.15.40.231 - 74 total, throughout 129.15.40.232 - 64082 total, throughout 129.15.40.231-rev - outsiders, few 129.15.40.232-rev - 90, all but 8 from 232

The latency node appears to make the most requests. We also have logs from both servers available:

http://packrat.internet2.edu/~aaron/karthik_logs/

To test I would suggest the following:

Set up a latency box (w/ 3.1.2), test to all USATLAS locations, see the following list:

BNL - lhcmon.bnl.gov BU - atlas-npt1.bu.edu UC - uct2-net1.uchicago.edu IU - iut2-net1.iu.edu UTA - netmon1.atlas-swt2.org OU - ps1.ochep.ou.edu MSU - psmsu01.aglt2.org UM - psum01.aglt2.org WISC - atlas-perfsonar1.chtc.wisc.edu SMU - i2ps1.physics.smu.edu NERSC - perfsonar.nersc.gov LBL - ettest.lbl.gov

Point resolv.conf at a specific DNS, and alert the maintainer of what we are doing. Let run for 48 or so hours, ask for logs.

It would be nice to see which tool is doing this, but if this is a latency node problem we can narrow it down to:

• owamp
• pinger
• pSB
• LS
• perfAdmin graphs

Assigning to Andy initially since he as access to LBL machines.

[arlake228]

Comment #1 originally posted by arlake228 on 2010-02-25T15:38:27.000Z:

My tentative thoughts are that this is a combination of owamp and pSB (mainly the former).

There was a bug fixed in the last release that would cause owamp to go into an 'connect' loop when tests were denied. Since owamp was using getaddrinfo, it's likely that this would have caused a ton of DNS queries. We actually saw something similar (lots of DNS queries) during SC, and when we turned off the testing and/or fixed the test denied problem, the queries would drop substantially.

All told, it's possible there are a substantial number of lookups even with the above fix included.

Other things that might increase the # of lookups:

powstream will retry any failed connection attempts twice every 10 seconds (two separate connections with a 10 second delay each averages out to about twice every 10 seconds). If there are enough servers denying tests or what-have-you, that could add up, though it shouldn't add too much.

powstream calls getnameinfo on the sender/receiver when generating the summary statistics to grab both the hostname and the IP for each.

One interesting thing to try would be to have the pSB master and pSB collector run on different machines, and collect DNS stats for each machine.

[arlake228]

Comment #2 originally posted by arlake228 on 2010-02-25T15:47:04.000Z:

The other item worth mentioning is what effect a caching DNS would have on this process:

pdnsd: http://www.debian-administration.org/articles/390

dnsmasq: http://www.thekelleys.org.uk/dnsmasq/doc.html

Could also do it the old fashioned way with bind.

If we are getting burned on the reverse especially, this may help.

[arlake228]

Comment #3 originally posted by arlake228 on 2010-02-25T15:52:12.000Z:

ps-bw.es.net and ps-lat.es.net are already hitting most of the boxes in that list. tcpdump shows a constant stream of DNS packets going out of the box where ps-bw has almost none (i.e. I can run tcpdump for a few minutes and not see any DNS queries).

Joe commented that this looks similar to previous issues with owamp. We also discussed the DNS caching approach which I see Jason just commented on as well. That might be a viable solution if we can't alter the code somehow to circumvent this.

[arlake228]

Comment #4 originally posted by arlake228 on 2010-02-25T18:09:57.000Z:

Another vote for caching (in addition to whatever we do to fix owamp) from BNL. I think they are more concerned about performance than the number of queries on their servers though.

[arlake228]

Comment #5 originally posted by arlake228 on 2010-03-01T13:51:53.000Z:

An update from OU:

Without yet doing all the data gathering, ps has done 1.7M dns lookups for 170 unique hostnames since the logs rolled on 129.15.1.20 at 0400 CST today. Interesting combo of numbers - I bet each hostname was requested 10k times. :)

Here is an excerpt:

26-Feb-2010 05:59:15.286 queries: client 129.15.40.231# 51572: query: iut2-net1.iu.edu IN AAAA + 26-Feb-2010 05:59:15.287 queries: client 129.15.40.231# 48641: query: iut2-net1.iu.edu IN AAAA + 26-Feb-2010 05:59:15.287 queries: client 129.15.40.231# 35601: query: iut2-net1.iu.edu IN A + 26-Feb-2010 05:59:15.288 queries: client 129.15.40.231# 33347: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:15.289 queries: client 129.15.40.231# 50238: query: 223.225.165.149.in-addr.arpa IN PTR + 26-Feb-2010 05:59:15.290 queries: client 129.15.40.231# 52277: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.209 queries: client 129.15.40.231# 38405: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:16.210 queries: client 129.15.40.231# 46392: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:16.210 queries: client 129.15.40.231# 46725: query: i2ps1.physics.smu.edu IN A + 26-Feb-2010 05:59:16.211 queries: client 129.15.40.231# 57205: query: ps1.ochep.ou.edu IN AAAA + 26-Feb-2010 05:59:16.212 queries: client 129.15.40.231# 50182: query: ps1.ochep.ou.edu IN AAAA + 26-Feb-2010 05:59:16.213 queries: client 129.15.40.231# 36248: query: ps1.ochep.ou.edu IN A + 26-Feb-2010 05:59:16.213 queries: client 129.15.40.231# 39905: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.240 queries: client 129.15.40.231# 49165: query: 190.200.119.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.358 queries: client 129.15.40.231# 39082: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:16.358 queries: client 129.15.40.231# 46207: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:16.359 queries: client 129.15.40.231# 48824: query: i2ps1.physics.smu.edu IN A + 26-Feb-2010 05:59:16.360 queries: client 129.15.40.231# 48269: query: ps1.ochep.ou.edu IN AAAA + 26-Feb-2010 05:59:16.361 queries: client 129.15.40.231# 56265: query: ps1.ochep.ou.edu IN AAAA + 26-Feb-2010 05:59:16.361 queries: client 129.15.40.231# 33368: query: ps1.ochep.ou.edu IN A + 26-Feb-2010 05:59:16.362 queries: client 129.15.40.231# 33521: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.389 queries: client 129.15.40.231# 43229: query: 190.200.119.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.509 queries: client 129.15.40.231# 55401: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.510 queries: client 129.15.40.231# 48484: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:16.510 queries: client 129.15.40.231# 49181: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:16.511 queries: client 129.15.40.231# 48381: query: i2ps1.physics.smu.edu IN A + 26-Feb-2010 05:59:16.512 queries: client 129.15.40.231# 39012: query: 190.200.119.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.513 queries: client 129.15.40.231# 32802: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:16.513 queries: client 129.15.40.231# 55789: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.245 queries: client 129.15.40.231# 56283: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.246 queries: client 129.15.40.231# 47733: query: 223.225.165.149.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.290 queries: client 129.15.40.231# 55650: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.291 queries: client 129.15.40.231# 38183: query: 223.225.165.149.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.787 queries: client 129.15.40.231# 55557: query: 231.40.15.129.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.787 queries: client 129.15.40.231# 47883: query: 106.164.75.207.in-addr.arpa IN PTR + 26-Feb-2010 05:59:20.795 queries: client 129.15.40.231# 41662: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:20.796 queries: client 129.15.40.231# 41874: query: i2ps1.physics.smu.edu IN AAAA + 26-Feb-2010 05:59:20.796 queries: client 129.15.40.231# 54200: query: i2ps1.physics.smu.edu IN A +

[arlake228]

Comment #6 originally posted by arlake228 on 2010-03-03T15:21:58.000Z:

Aaron and I installed dnscache (http://cr.yp.to/djbdns/dnscache.html) on ps-lat and an Internet2 host and that seems to have helped a lot. It was as easy as running 'apt-get install nscd' and using the default configuration. Outgoing DNS queries went to an average of ~20 packets/sec to something more reasonable (i.e. last time i ran it for 60 seconds I saw a total of 10 DNS packets).

Aaron and I also added some debugging output to the owamp code and ran that on ps-lat. Looking at the code it looks like owamp is calling getaddrinfo in multiple places that's likely causing the flood of DNS queries. It looks like it would be fairly painful to reorganize those in a way that wouldn't break things in some other way so the dnscache solution might be the best way forward for awhile. We should probably include this on the next disk. We could also tell sites that are having problems to run apt-get and restart their tests.

[arlake228]

Comment #7 originally posted by arlake228 on 2010-03-03T15:27:17.000Z:

How many days did you have the test cache running and would you favor I recommend this to OU as a test case?

Is there any additional configuration beyond installing it (e.g. messing with /etc/resolv.conf or editing a new configuration file)?

[arlake228]

Comment #8 originally posted by arlake228 on 2010-03-03T15:29:49.000Z:

Correction: I looked at too many of these packages yesterday. ncsd is actually it own project (http://linux.die.net/man/8/nscd). Was thinking dnscache project was what produced this but was wrong. The aptget command was correct.

[arlake228]

Comment #9 originally posted by arlake228 on 2010-03-03T15:50:13.000Z:

At this point its been running for less than 24 hours. The nice thing about nscd is that it does not require any changes to resolv.conf. It can do this because glibc is designed to check for the existence of nscd when things like gethostbyname get called. If a name is not in the cache yet, it will use resolv.conf like normal then cache it (Aaron can correct me if i misstated anything there, but that should be the high-level idea).

As such, it also appears to play nicely with dhclient. Aaron installed it on lab253 which has DHCP configured and it seemed to work fine. Most of the other packages would have required some scripts to mess with resolv.conf if dhcp was configured, which is why we tried this.

[arlake228]

Comment #10 originally posted by arlake228 on 2010-03-03T16:09:55.000Z:

In response to the last part of Jason's post, yes I would favor having OU try this. We can maybe give it until the end of the day just to be on the safe side. To install, it's literally 'apt-get install nscd' then restart the tests. No additional configuration files to edit. The default settings seem to work out-of-the-box.

[arlake228]

Comment #11 originally posted by arlake228 on 2010-03-03T16:11:33.000Z:

OU was notified, I will follow up on this with some more data when I have it.

[arlake228]

Comment #12 originally posted by arlake228 on 2010-03-03T18:47:30.000Z:

In response to comment # 6:

Andy, getaddrinfo is only called from within the I2Addr functions in the I2Utils library, so it would not be difficult to add some kind of cache in the daemon itself. That said, I don't think it is appropriate except in some exceptional cases. (misses for example.) owampd should not be trying to determine the appropriate TTL for dns entries, there are resolver/dns caches for that. In my opinion, the caching dns server is the appropriate solution here.

[arlake228]

Comment #13 originally posted by arlake228 on 2010-03-05T04:57:52.000Z:

I do agree with the sentiment here. BIND would be a more dns specific and configurable solution. But... would that require more configuration?

Hi Jeff,

I think there was some mis-communication going on here.

Here's a followup email which will hopefully clear up the confusion between nscd and bind.

Thanks,

Horst

---

From: "Laws, Peter C." plaws@ou.edu To: "Severini, Horst" hs@nhn.ou.edu, "George, Brandon C." bcg@ou.edu CC: "karunach@nhn.ou.edu" karunach@nhn.ou.edu Subject: RE: Fwd: ps1.ochep.ou.edu? Date: Thu, 4 Mar 2010 22:09:47 +0000

That explains why there are still requests, then.

If it's like the old Solaris Name Services Caching Daemon, it just does a small local cache of recently queried names. In Solaris, it worked for NIS as well.
The problem with it was (on Solaris) is that names would legitimately change in DNS and clients would still see old info. On Suns, I think the command to clear the cache was "nscd -i hosts" or something.

If the nscd doesn't work out, a copy of BIND running in "caching only" mode may give better results. Caching servers know about the root name servers and nothing else - no zones, not even slaves -- and queries for everything else.
You end up with a nice cache of frequently-requested names.

Peter

Peter Laws / N5UWY National Weather Center / Network Operations Center / Web University of Oklahoma Information Technology plaws@ou.edu

---

From: Horst Severini [hs@nhn.ou.edu] Sent: Thursday, March 04, 2010 15:55 To: Severini, Horst; George, Brandon C.; Laws, Peter C. Cc: karunach@nhn.ou.edu Subject: RE: Fwd: ps1.ochep.ou.edu?

Hi Peter,

that sounds a lot better indeed!

We installed nscd, so that wouldn't be expected to completely replace a DNS server, or would it? I don't know much about it, just guessing ...

Thanks a lot,

   Horst

[arlake228]

Comment #14 originally posted by arlake228 on 2010-03-05T13:28:27.000Z:

nscd has the benefit of install and forget. With bind, we'd have to replace the resolv.conf with a pointer to the localhost, and work around the fact that dhclient likes to write that file.

We could decrease the requests by upping how long nscd caches the data. There are some configuration options for DNS hits and misses. We could up the DNS hits to 2 or 3 hours from the current 1, and up misses from 20 seconds to something higher. I'm not sure what all would be affected by this nor why the nscd folks chose the numbers they did, but I'd prefer to not mess with the defaults without good reason.

My guess is that the bulk of the queries are coming from cache.pl/pinger_cache.pl which run once an hour or so. If we backed that off to once a day, or a few times a day, we could decrease those requests.

[arlake228]

Comment #15 originally posted by arlake228 on 2010-03-05T13:57:22.000Z:

Which dhcp client are we using? (Sorry, I should probably know this...) But, if we are using dhclient would make this pretty easy to augment resolv.conf. /etc/dhcp3/dhclient.conf prepend domain-name-servers 127.0.0.1;

If our dhcp client does not make that easy...

I think there is good reason to increase the cache misses from 20 seconds to something higher.

Do you know if the DNS TTL is paid attn to in nscd? i.e. If there is a hit, I would think it should be cached a for a max of MIN(nscd cache max, dns record ttl). If it works this way, increasing the nscd cache max would be a good idea. If not... then we should be careful with how we do this. But, in the end I do think the dns record ttl should be used. If nscd doesn't do that, we should figure out a better way.

[arlake228]

Comment #16 originally posted by arlake228 on 2010-03-05T14:18:06.000Z:

We're using dhclient. It's easy enough to augment resolv.conf but we also need to get the nameservers returned by DHCP into named.conf for bind. This would require a custom script of some sort which is doable but extra overhead compared to just installing nscd.

Also, nscd does pay attention to DNS TTL...mostly. See post here from one of the glibc maintainers: http://udrepper.livejournal.com/16362.html. There's some conflicting information out there if you look around because I think it only acknowledges it for calls to getaddrinfo (which owamp uses) but doesn't for gethostbyname.

[arlake228]

Comment #17 originally posted by arlake228 on 2010-03-12T18:08:22.000Z:

A huge part of this problem is the dns misses. As long as it works to cache those in nscd, and something more reasonable like 20 minutes - then I think this is a fine solution.

[arlake228]

Comment #18 originally posted by arlake228 on 2010-03-12T19:13:40.000Z:

Targeting for next release.

[arlake228]

Comment #19 originally posted by arlake228 on 2010-03-12T21:27:40.000Z:

You can change the amount of time it caches misses to 20 minutes by setting the following in /etc/nscd.conf:

negative-time-to-live hosts 1200

We should be able to put that in the default config file we ship with the disk. I put it on ps-lat and ran tcpdump for about 5 minutes and saw one DNS request so I think it works as advertised.

[arlake228]

Comment #20 originally posted by arlake228 on 2010-03-19T15:29:35.000Z:

I've added nscd.conf to the repository, and bumped the negative ttl to 20 minutes. The disk has been updated.

[arlake228]

Comment #21 originally posted by arlake228 on 2010-03-25T18:50:39.000Z:

closing

[arlake228]

Comment #22 originally posted by arlake228 on 2011-11-01T20:12:15.000Z:

<empty>