search

qmsk / clusterf

Clustered IPVS load balancer control plane with Docker integration
MIT License
35 stars 4 forks source link

Remove failed clusterf-docker hosts #1

Closed SpComb closed 8 years ago

SpComb commented 8 years ago

The clusterf-docker will retract the ConfigServiceBackend for a container that is stopped, but the backends are not cleaned up if the the docker host itself dies.

This needs some timeout and/or health-checking mechanism.

SpComb commented 8 years ago

The new clusterf-docker --etcd-timeout=10s default writes etcd routes/services with a 10s TTL / 5s refresh per default, such that a dead clusterf-docker will expire said services. clusterf-docker will also immediately cleanup before exiting if docker dies, or it is killed.

$ clusterf-docker ...
2016/05/09 16:51:04 docker:Docker.Listen...
2016/05/09 16:51:04 Update config...
2016/05/09 16:51:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:51:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:51:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new routes/37334d3c1c5ca7239f9fafce6937912913342997efceaaa5089b8196d1f7d95a
2016/05/09 16:51:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:51:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b

$ clusterf-ipvs ...
2016/05/09 16:51:05 config:EtcdSource.watch: set services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
2016/05/09 16:51:05 config:EtcdSource.watch: set services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:51:05 IPVS: New service inet+udp://10.0.107.1:25826
2016/05/09 16:51:05 IPVS: New service inet+udp://10.0.107.1:25826 dest 10.6.107.2:25826
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                      10.6.107.2:25826 masq
2016/05/09 16:51:05 config:EtcdSource.watch: set routes/37334d3c1c5ca7239f9fafce6937912913342997efceaaa5089b8196d1f7d95a
2016/05/09 16:51:05 IPVS: New service inet+udp://10.0.107.1:25826 dest 10.107.107.6:25826
2016/05/09 16:51:05 IPVS: Delete service inet+udp://10.0.107.1:25826 dest 10.6.107.2:25826
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:05 config:EtcdSource.watch: set services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:05 config:EtcdSource.watch: set services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:09 config:EtcdSource.watch: set services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:10 config:EtcdSource.watch: set routes/37334d3c1c5ca7239f9fafce6937912913342997efceaaa5089b8196d1f7d95a
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:10 config:EtcdSource.watch: set services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:10 config:EtcdSource.watch: set services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute
2016/05/09 16:51:10 config:EtcdSource.watch: set services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
udp                       10.0.107.1:25826 wlc
                    10.107.107.6:25826 droute

Forcefully killing clusterf-docker without giving it a chance to cleanup:

$ killall -9 clusterf-docker

Results in:

2016/05/09 16:51:20 config:EtcdSource.watch: expire services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:51:20 IPVS: Delete service inet+udp://10.0.107.1:25826
Proto                           Addr:Port
2016/05/09 16:51:20 config:EtcdSource.watch: expire routes/37334d3c1c5ca7239f9fafce6937912913342997efceaaa5089b8196d1f7d95a
Proto                           Addr:Port
2016/05/09 16:51:20 config:EtcdSource.watch: expire services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
2016/05/09 16:51:20 config:EtcdSource.watch: expire services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port
2016/05/09 16:51:20 config:EtcdSource.watch: expire services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
Proto                           Addr:Port