weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 671 forks source link

[dns] stops working after a while #262

Closed yaronr closed 9 years ago

yaronr commented 9 years ago

3 ec-2 CoreOS nodes, connected via weave. weave status is fine, ping from all nodes to all nodes work. 2 instances of elasticsearch, 1 instance of logstash.

Logstash container has the ip: 10.100.0.2 Elasticsearch: 10.100.0.5, 10.100.0.7

INFO: 2014/12/05 09:57:13.020910 Using mDNS on ethwe INFO: 2014/12/05 09:57:13.021479 Listening for DNS on :53 INFO: 2014/12/05 09:57:13.356243 Ignoring name 6fe25a31bf94.., not in weave.local. INFO: 2014/12/05 09:57:14.051802 Ignoring name 9db1aa1cff22.., not in weave.local. INFO: 2014/12/05 09:57:14.178479 Adding logstash.weave.local. -> 10.100.0.11 INFO: 2014/12/05 09:57:14.188226 Adding logstash.weave.local. -> 10.100.0.3 INFO: 2014/12/05 09:57:14.197944 Adding logstash.weave.local. -> 10.100.0.2 INFO: 2014/12/05 09:57:14.327735 Adding elasticsearch.weave.local. -> 10.100.0.6 INFO: 2014/12/05 09:57:14.337320 Adding elasticsearch.weave.local. -> 10.100.0.1 INFO: 2014/12/05 09:57:14.347277 Adding elasticsearch.weave.local. -> 10.100.0.5

bboreham commented 9 years ago

Did you edit this? The email notification I got had a lot longer logs, but not the "Adding logstash.weave.local." part.

As it stands I can't figure out what the issue is.

yaronr commented 9 years ago

Yes,

I dug a bit deeper and figured out there are stale DNS entries

Regards, Yaron Rosenbaum Founder and CTO MultiCloud M +972 54 2346012 www.multicloud.io http://www.multicloud.io/

On Dec 5, 2014, at 12:57 PM, Bryan Boreham notifications@github.com wrote:

Did you edit this? The email notification I got had a lot longer logs, but not the "Adding logstash.weave.local." part.

As it stands I can't figure out what the issue is.

— Reply to this email directly or view it on GitHub https://github.com/zettio/weave/issues/262#issuecomment-65774881.

squaremo commented 9 years ago

Those initial log entries (from the original log) suggest there are more logstash and elasticsearch containers attached than you think there are -- this either means there are more containers running, or those that are running are (or look) attached to the weave network more than once. weave ps gives you the same data as weavedns sees when it populates its DB, so you could look at that.

I also see no entries in the log showing containers being detached or containers exiting. Should there be?

There's no entry for logstash being added, so I presume that's on another host?

rade commented 9 years ago

@yaronr is this problem still occurring?

yaronr commented 9 years ago

Hi @rade Sorry, I missed your question. No, I think it's not. Thanks