search

AdRoll / hologram

Easy, painless AWS credentials on developer laptops.
Apache License 2.0
803 stars 42 forks source link

Occasional DNS lookup failures when calling hologram agent #73

Open copumpkin opened 8 years ago

copumpkin commented 8 years ago

I'm not sure what would cause this, but I've seen this a few times on more than one computer:

$ hologram use somerole
[ERROR  ] 2016-02-22T09:04:09-05:00 dial tcp: lookup hologram.server.dns: no such host

If I try pinging hologram.server.dns (not the real domain) myself, everything else on my system seems to be able to resolve it just fine. Hologram will continue complaining about the "no such host" for a while and then after a few minutes, will start resolving it fine again.

It might worth noting that hologram.server.dns started as a cname for an Amazon ELB. After seeing this issue for a bit, I pointed my hologram agent directly at the ELB DNS, in case hologram didn't like cnames, but the issue persists.

Any suggestions? I don't see anything particularly interesting in the agent logs, but if someone has suggestions on how to debug this I'm happy to try things next time it happens.

copumpkin commented 8 years ago

I'd normally use ltrace here, but I think that's less useful with Go programs.

aebie commented 8 years ago

I have been having the same issue intermittently. Until today it generally cleared up in a short 10 minutes or so, but I have been getting the error for about 3 hours straight now.

copumpkin commented 8 years ago

I am on Mac OS when this happens, and https://golang.org/pkg/net/#pkg-overview suggests there might be some intricacies in ho Go does DNS on Mac OS.

copumpkin commented 8 years ago

Also potentially relevant: http://stackoverflow.com/questions/12952833/lookup-host-no-such-host-error-in-go and https://github.com/golang/go/issues/3575.

copumpkin commented 8 years ago

I've noticed that I get the issue when I switch roles in rapid succession. Anyone else get this?