vertx-service-discovery-bridge-kubernetes - Scale down of pods not working

[adasari]

Hi.. Rest API calls getting timedout when prods are scaled down. Seems request has been routed a member which is not a member any more.

Thanks.

[cescoffier]

Hi,

Can you try with Hazelcast 3.7.1. Add:

<dependency>
      <groupId>com.hazelcast</groupId>
      <artifactId>hazelcast</artifactId>
      <version>3.7.1</version>
</dependency>

as a dependency to your project.

I've tried with this version and it seems to work (I had issues with 3.6.1)

[adasari]

Sure. i will test and let you know. Thank you very much.

[cescoffier]

I've configured Hazelcast as follows:

    <property name="hazelcast.health.monitoring.delay.seconds">2</property>

    <property name="hazelcast.max.no.heartbeat.seconds">20</property>
    <property name="hazelcast.max.no.master.confirmation.seconds">30</property>
    <property name="hazelcast.master.confirmation.interval.seconds">10</property>
    <property name="hazelcast.member.list.publish.interval.seconds">10</property>
    <property name="hazelcast.connection.monitor.interval">10</property>
    <property name="hazelcast.connection.monitor.max.faults">2</property>
    <property name="hazelcast.partition.migration.timeout">10</property>
    <property name="hazelcast.migration.min.delay.on.member.removed.seconds">3</property>

[adasari]

i tried with following pom.xml

    <dependency>
        <groupId>io.vertx</groupId>
        <artifactId>vertx-hazelcast</artifactId>
        <version>3.3.3</version>
        <exclusions>
            <exclusion>
                <groupId>com.hazelcast</groupId>
                <artifactId>hazelcast</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
           <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast</artifactId>
        <version>3.7.1</version>
    </dependency>
           <dependency>
        <groupId>io.vertx</groupId>
        <artifactId>vertx-service-discovery-bridge-kubernetes</artifactId>
        <version>3.3.3</version>
    </dependency>

and cluster.xml disconver strategies is

    <discovery-strategy enabled="true"
        class="io.vertx.servicediscovery.hazelcast.HazelcastKubernetesDiscoveryStrategy">
  <properties>            
        <property name="service-name">test-name</property>
         <property name="service-label-name">test-label</property>
          <property name="service-label-value">true</property>
        <property name="namespace">default</property>
      </properties>
    </discovery-strategy>

scale up did not join the cluster. all nodes have thier own cluster with one node. this is strange. i will see if i did any mistake and retest. thanks.

[adasari]

No hazelcast 3.7.1 with vertex 3.3.3 is not working. Scale up also not joining the cluster. what is the vertx version you used for your testing ? can you have sample project ?

i reverted to original pom.xml (vertx 3.3.3 which uses hazelacst 3.6.3) then scale up worked.

Am i doing something wrong here? Thanks.

[cescoffier]

Here: https://github.com/vert-x3/vertx-examples/tree/openshift-hazelcast-test

But I'm actually using a different version of the Hazelcast discovery for Kubernetes:

https://github.com/vert-x3/vertx-service-discovery/tree/kubernetes-hazelcast

[adasari]

Should I test using vertx-service-discovery-3.4.0 with vertx-3-3.3 ? or all with 3.4.0 ? please clarify.

[cescoffier]

With 3.4.0-SNAPSHOT, however, as the changes are not merged yet, you would need to check out the code, switch to the branch and build them:

For the discovery:

git clone https://github.com/vert-x3/vertx-service-discovery.git
cd vertx-service-discovery
git checkout kubernetes-hazelcast
mvn clean install -DskipTests

For the examples

git clone https://github.com/vert-x3/vertx-examples.git
cd vertx-examples
git checkout openshift-hazelcast-test
cd openshift3-examples
mvn clean package

[adasari]

sure. thanks. i will test it.