Automatic discovery of all nodes in namespace causes trouble with 2 or more separate HZ clusters

[fiksn]

Hi,

with 0.9.3 version I saw that resolve() was made "smarter" (https://github.com/noctarius/hazelcast-kubernetes-discovery/blob/master/src/main/java/com/noctarius/hazelcast/kubernetes/ServiceEndpointResolver.java#L68). But in my case that is quite a problem.

We have two separate hazelcast clusters on kubernetes (in same namespace). Let's say I start one of them. Two pods come up and I'd expect they will eventually form a cluster (but until hz is intialized readiness check prevents one to see the other node). That means I expect the discovery plugin to say there are no other nodes. What happens instead is that nodes from the other cluster are discovered but credentials are incorrect so at the end hz can't connect anywhere and just terminates.

2016-11-11 13:40:13,247 [localhost-startStop-1] [DEBUG] [c.h.i.c.impl.DiscoveryJoiner] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] PostJoin master: null, isMaster: false
2016-11-11 13:40:13,248 [localhost-startStop-1] [ERROR] [com.hazelcast.instance.Node] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] Could not join cluster in 300000 ms. Shutting down now!
2016-11-11 13:40:13,250 [localhost-startStop-1] [INFO ] [c.h.core.LifecycleService] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] [10.201.36.12]:5701 is SHUTTING_DOWN
2016-11-11 13:40:13,251 [localhost-startStop-1] [WARN ] [com.hazelcast.instance.Node] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] Terminating forcefully...

Can this additional logic to discover nodes be disabled somehow?

[amoAHCP]

Hi, you can separate them by using service labels like this: https://github.com/amoAHCP/kube_vertx_demo/blob/discovery-bridge-kubernetes/kube/frontend-service.yaml. How is your setup for the clusters? Does each cluster have an own Service?

[fiksn]

Yes, I have a different Service for each cluster. But the problem is that "kubectl get endpoints" returns nothing due to "readiness check". Only if hazelcast was successfully initialized a /healthz endpoint of my webapp would be set-up and return HTTP 200. And then the Pod ip would be available as Endpoint. So this is sort of a chicken-and-egg issue.

I am not using service labels (yet) but also if I did it there is still this: https://github.com/noctarius/hazelcast-kubernetes-discovery/blob/master/src/main/java/com/noctarius/hazelcast/kubernetes/ServiceEndpointResolver.java#L79

This will mistakenly "discover" the "other" hazelcast cluster and cause everything to fail.

Sometimes it is better to not discover too much. In the mentioned case what I'd want is (sorry if I am a bit too verbose):

• first cluster is running (multiple Pods created through Rc and one Service to route traffic)
• I start first Pod of the other (second) cluster (Service is there but empty)
• hazelcast discovery should fail (and NOT discover anything on first cluster)
• Pod should start normally (as master) and advertise itself through Service after health-check is ok
• second Pod can then correctly discover that first node through the Service

[fiksn]

Either that or hazelcast itself should be aware that the "discovered" nodes have a different hazelcast/group/name setting and thus don't "count" and the node should still (eventually) become master.

In hazelcast-kubernetes-discovery I see a simple solution with a new settings (possibly defaulting to true) whether you want to fallback to ANY node in namespace.

[noctarius]

Maybe the NodeFilter (http://docs.hazelcast.org/docs/3.7/javadoc/com/hazelcast/spi/discovery/NodeFilter.html) is what you're looking for. With a NodeFilter being configured you can add additional selection logic to the discovery.

Apart from that, the labels are exactly what would be setup to groupname for example and could be used for filtering in the plugin itself as @JacpFX mentioned.

[fiksn]

Thanks for the NodeFilter hint. So basically I could configure DiscoveryNode properties through the XML configuration and pass my implementation of NodeFilter to match by a certain key/value pair and thus "disqualifying" the members of the other cluster?

I wonder why the kubernetes discovery plugin then even goes through the trouble of using serviceName / serviceLabels, it could just do getNodesByNamespace() all the time. Is it only for performance reasons?

[noctarius]

Because NodeFilter is part of the SPI and works for all discovery plugins while labels are part of kubernetes and most of the time you want to use the native possibilities.

[st0ne-dot-at]

may fixed by #33 . @fiksn : may you can verify that?

[xqianwang]

@st0ne-dot-at that doesn't work

[googlielmo]

Hi @fiksn I'm closing this for now, as with #30 you can separate your clusters by using service labels, regardless of readiness checks.