search

vert-x3 / vertx-hazelcast

Hazelcast Cluster Manager for Vert.x
Apache License 2.0
78 stars 75 forks source link

Microservices sometimes doesn't discover Services over Kubernetes #87

Closed shaykh-salman closed 6 years ago

shaykh-salman commented 6 years ago

Hi,

I'm trying to build some microservices which are using vertx-hazelcast library and I'm deploying it over kubernetes on AWS EC-2 instance. The microservices sometimes get disconnected from hazelcast cluster and sometimes are not discoverable. If there anything I'm missing ? Need some advice.

My architecture

Fat Jar -> Docker Image -> Kubernetes (AWS EC-2)

tsegismont commented 6 years ago

Can you provide more context? Hazelcast cluster.xml file? Log snippets? Details about the disconnection.

shaykh-salman commented 6 years ago

Hi,

I moved over to digital ocean and I was trying to do the service discovery by TCP/IP since digital ocean doesn't support multicast. Please Have a look at my cluster.xml file

`<?xml version="1.0" encoding="UTF-8"?>

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.2.xsd" xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

false false false 0 jdk 
<network>
    <join>
        <multicast enabled="false">
            <multicast-group>224.2.2.3</multicast-group>
            <multicast-port>54327</multicast-port>
        </multicast>
        <tcp-ip enabled="true">
            <interface>archer-fetchsome-product-microservice</interface>
        </tcp-ip>
    </join>
    <interfaces enabled="true">
        <interface>10.*.*.*</interface>
    </interfaces>
</network>
<semaphore name="__vertx.*">
    <initial-permits>1</initial-permits>
</semaphore>

`

`Feb 02, 2018 2:06:01 AM io.vertx.core.impl.launcher.commands.RunCommand INFO: Starting clustering... Feb 02, 2018 2:06:01 AM io.vertx.core.impl.launcher.commands.RunCommand INFO: No cluster-host specified so using address 10.2.1.31 Feb 02, 2018 2:06:01 AM com.hazelcast.config.AbstractXmlConfigHelper WARNING: Name of the hazelcast schema location incorrect using default Feb 02, 2018 2:06:02 AM com.hazelcast.instance.DefaultAddressPicker WARNING: [LOCAL] [dev] [3.8.2] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct. Feb 02, 2018 2:06:02 AM com.hazelcast.instance.DefaultAddressPicker INFO: [LOCAL] [dev] [3.8.2] Resolving domain name 'archer-fetchsome-apigw-microservice' to address(es): [10.3.39.164] Feb 02, 2018 2:06:02 AM com.hazelcast.instance.DefaultAddressPicker INFO: [LOCAL] [dev] [3.8.2] Interfaces is enabled, trying to pick one address matching to one of: [archer-fetchsome-apigw-microservice/10...*] Feb 02, 2018 2:06:02 AM com.hazelcast.instance.DefaultAddressPicker INFO: [LOCAL] [dev] [3.8.2] Prefer IPv4 stack is true. Feb 02, 2018 2:06:02 AM com.hazelcast.instance.DefaultAddressPicker INFO: [LOCAL] [dev] [3.8.2] Picked [archer-fetchsome-apigw-microservice]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true Feb 02, 2018 2:06:02 AM com.hazelcast.system INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Hazelcast 3.8.2 (20170518 - a60f944) starting at [archer-fetchsome-apigw-microservice]:5701 Feb 02, 2018 2:06:02 AM com.hazelcast.system INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. Feb 02, 2018 2:06:02 AM com.hazelcast.system INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Configured Hazelcast Serialization version : 1 Feb 02, 2018 2:06:02 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Backpressure is disabled Feb 02, 2018 2:06:03 AM com.hazelcast.instance.Node INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Creating TcpIpJoiner Feb 02, 2018 2:06:04 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Starting 2 partition threads Feb 02, 2018 2:06:04 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Starting 3 generic threads (1 dedicated for priority tasks) Feb 02, 2018 2:06:04 AM com.hazelcast.core.LifecycleService INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] [archer-fetchsome-apigw-microservice]:5701 is STARTING Feb 02, 2018 2:06:04 AM com.hazelcast.nio.tcp.InitConnectionTask INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Connecting to archer-fetchsome-apigw-microservice/10.3.39.164:5702, timeout: 0, bind-any: true Feb 02, 2018 2:06:04 AM com.hazelcast.nio.tcp.InitConnectionTask INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Connecting to archer-fetchsome-apigw-microservice/10.3.39.164:5703, timeout: 0, bind-any: true Feb 02, 2018 2:06:09 AM com.hazelcast.system INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Cluster version set to 3.8 Feb 02, 2018 2:06:09 AM com.hazelcast.cluster.impl.TcpIpJoiner INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2]

Members [1] { Member [archer-fetchsome-apigw-microservice]:5701 - 4f217632-a431-487c-a900-c85823dcadb6 this }

Feb 02, 2018 2:06:09 AM com.hazelcast.core.LifecycleService INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] [archer-fetchsome-apigw-microservice]:5701 is STARTED Feb 02, 2018 2:06:09 AM com.hazelcast.internal.partition.impl.PartitionStateManager INFO: [archer-fetchsome-apigw-microservice]:5701 [dev] [3.8.2] Initializing cluster partition table arrangement... Feb 02, 2018 2:06:09 AM io.vertx.core.impl.HAManager INFO: A quorum has been obtained. Any deploymentIDs waiting on a quorum will now be deployed Feb 02, 2018 2:06:10 AM com.fairbit.extension.tool.ApiGateWay INFO: API Gateway is running on port 8080host archer-fetchsome-apigw-microservice Feb 02, 2018 2:06:10 AM io.vertx.core.impl.launcher.commands.VertxIsolatedDeployer INFO: Succeeded in deploying verticle {"location":{"endpoint":"https://archer-fetchsome-apigw-microservice:8080/","host":"archer-fetchsome-apigw-microservice","port":8080,"root":"/","ssl":true},"metadata":{},"name":"api-gateway","registration":"7416d5ab-3d3b-4102-a8b7-9f0c312689c3","status":"UP","type":"api-gateway"} Service published. Feb 02, 2018 2:06:10 AM com.fairbit.commonservices.commonservices.BaseVerticle INFO: Service published`

tsegismont commented 6 years ago

For static IP discovery, see http://docs.hazelcast.org/docs/3.8.2/manual/html-single/index.html#discovering-members-by-tcp

Looking at your cluster XML file, I would say that you are missing a list of <member> tags in the <tcp> section.

Please follow up on the user group as it seems it's more a getting started issue than a bug. Thank you.