search

hseeberger / constructr

Coordinated (etcd, ...) cluster construction for dynamic (cloud, containers) environments
Apache License 2.0
212 stars 37 forks source link

#159 filter selfnode from nodeslist #162

Closed jasongoodwin closed 7 years ago

jasongoodwin commented 7 years ago

Re: https://github.com/hseeberger/constructr/issues/159

This will filter the selfnode from the list of nodes received and warn if it's encountered.

I'm not sure if this is the prettiest approach but I tested and it does work. This is a fairly easy scenario to produce. Using a fixed ip address, If we shutdown the old nodes and then try to start up again nodes can see the selfnode while other nodes are acquiring the lock and trying to start up

[WARN][17:43:10.907][d.h.c.ConstructrMachine][C:] - Selfnode received in list of nodes Set(akka.tcp://testcartCluster@10.102.3.6:2756). Will filter to prevent forming an island.
[WARN][17:43:10.908][d.h.c.ConstructrMachine][C:] - Couldn't acquire lock, going to GettingNodes
[WARN][17:43:10.909][d.h.c.ConstructrMachine][C:] - Selfnode received in list of nodes Set(akka.tcp://testcartCluster@10.102.3.6:2756). Will filter to prevent forming an island.
[WARN][17:43:10.910][d.h.c.ConstructrMachine][C:] - Couldn't acquire lock, going to GettingNodes

The filtering is done in the retrieve section which is significant - the node will be filtered before determining if there are nodes or not. The implication is that if only the self node exists, it will attempt to lock if the only node is selfNode as it would be filtered.

hseeberger commented 7 years ago

This is another great example for a nasty corner case when we allow the cluster to be formed more than once.