kelseyhightower / confd

Manage local application configuration files using templates and data from etcd or consul
MIT License
8.34k stars 1.41k forks source link

Intensive CPU Load after Zookeeper timed out #463

Open duilio opened 8 years ago

duilio commented 8 years ago

Apparently when confd loss the connection to ZK it tries to connect infinitely without succeeding.

I basically see two issues:

  1. confd is not able to restore the connection to ZK
  2. confd retries continuously raising the CPU usage

These are the logs:

...
2016/06/28 23:23:41 Recv loop terminated: err=read tcp 192.168.1.66:59717->192.168.1.83:2181: i/o timeout
2016/06/28 23:23:41 Send loop terminated: err=<nil>
2016/06/28 23:23:42 Failed to connect to 192.168.1.64:2181: dial tcp 192.168.1.64:2181: i/o timeout
2016/06/28 23:23:43 Connected to 192.168.1.82:2181
2016/06/28 23:23:43 Authenticated: id=672446297887049508, timeout=4000
2016/06/28 23:23:50 Recv loop terminated: err=read tcp 192.168.1.66:42979->192.168.1.82:2181: i/o timeout
2016/06/28 23:23:50 Send loop terminated: err=<nil>
2016/06/28 23:23:50 Connected to 192.168.1.35:2181
2016/06/28 23:23:50 Authenticated: id=672446297887049508, timeout=4000
2016/06/28 23:23:53 Recv loop terminated: err=read tcp 192.168.1.66:36424->192.168.1.35:2181: i/o timeout
2016/06/28 23:23:53 Send loop terminated: err=<nil>
2016/06/28 23:23:54 Failed to connect to 192.168.1.65:2181: dial tcp 192.168.1.65:2181: i/o timeout
2016/06/28 23:23:55 Failed to connect to 192.168.1.83:2181: dial tcp 192.168.1.83:2181: i/o timeout
2016/06/28 23:23:55 Connected to 192.168.1.64:2181
2016/06/28 23:23:55 Authenticated: id=672446297887049508, timeout=4000
2016/06/28 23:24:00 Recv loop terminated: err=read tcp 192.168.1.66:43166->192.168.1.64:2181: i/o timeout
2016/06/28 23:24:00 Send loop terminated: err=<nil>
...

When the service is restarted the connection is correctly restored and the CPU usage is regular.

We are using the current master branch of confd configured to connect to ZK using watchers.

hstarorg commented 7 years ago

same too.

HelloChenHZ commented 6 years ago

Is there any suggestions or temporary method for intensive CPU Load after Zookeeper timed out?