search

hazelcast / hazelcast-kubernetes

Kubernetes Discovery for Hazelcast
Apache License 2.0
173 stars 99 forks source link

Each pod is forming it is own cluster #108

Closed TockaAyman-VG closed 5 years ago

TockaAyman-VG commented 5 years ago

Hello all, I am trying to get hazelcast up and running on kubernetes cluster [I am using vertx framework in the app] , I am almost done except that each pod is forming it is own cluster. Can any one tell me what is wrong in my configuration, Thanks in advance

I added this in my gradle dependencies compile group: 'io.vertx', name: 'vertx-hazelcast', version: '3.5.4' which internally added com.hazelcast:hazelcast:3.8.2

This is my cluster.xml I added it in the resource folder in my app `<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.8.xsd" xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<properties>
    <property name="hazelcast.discovery.enabled">true</property>
</properties>

<network>
    <join>
        <!-- deactivate normal discovery -->
        <multicast enabled="false"/>
        <tcp-ip enabled="false" />
        <!-- activate the Kubernetes plugin -->
        <discovery-strategies>
            <discovery-strategy enabled="true"
                                class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
            </discovery-strategy></discovery-strategies>
    </join>
</network>

`

I added rbac.yaml

kind: ClusterRoleBinding
metadata:
  name: default-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: ServiceAccount
  name: default
  namespace: default

then deployed my deployment and service.

I also updated my docker file run command to use java -jar jarName -cluster

and here is the starting logs 

Oct 15, 2018 8:26:44 AM io.vertx.core.impl.launcher.commands.RunCommand
INFO: Starting clustering...
Oct 15, 2018 8:26:44 AM io.vertx.core.impl.launcher.commands.RunCommand
INFO: No cluster-host specified so using address 10.4.2.31
Oct 15, 2018 8:26:45 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.8.2] Prefer IPv4 stack is true.
Oct 15, 2018 8:26:45 AM com.hazelcast.instance.DefaultAddressPicker
INFO: [LOCAL] [dev] [3.8.2] Picked [10.4.2.31]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Oct 15, 2018 8:26:45 AM com.hazelcast.system
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Hazelcast 3.8.2 (20170518 - a60f944) starting at [10.4.2.31]:5701
Oct 15, 2018 8:26:45 AM com.hazelcast.system
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
Oct 15, 2018 8:26:45 AM com.hazelcast.system
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Configured Hazelcast Serialization version : 1
Oct 15, 2018 8:26:45 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Backpressure is disabled
Oct 15, 2018 8:26:46 AM com.hazelcast.instance.Node
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Activating Discovery SPI Joiner
Oct 15, 2018 8:26:46 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Starting 2 partition threads
Oct 15, 2018 8:26:47 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Starting 3 generic threads (1 dedicated for priority tasks)
Oct 15, 2018 8:26:47 AM com.hazelcast.core.LifecycleService
INFO: [10.4.2.31]:5701 [dev] [3.8.2] [10.4.2.31]:5701 is STARTING
Oct 15, 2018 8:26:47 AM com.hazelcast.system
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Cluster version set to 3.8
Oct 15, 2018 8:26:47 AM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [10.4.2.31]:5701 [dev] [3.8.2]

Members [1] {
        Member [10.4.2.31]:5701 - 025755f0-b3c4-4e51-8853-78939418e661 this
}

Oct 15, 2018 8:26:47 AM com.hazelcast.core.LifecycleService
INFO: [10.4.2.31]:5701 [dev] [3.8.2] [10.4.2.31]:5701 is STARTED
Oct 15, 2018 8:26:47 AM com.hazelcast.internal.partition.impl.PartitionStateManager
INFO: [10.4.2.31]:5701 [dev] [3.8.2] Initializing cluster partition table arrangement...
leszko commented 5 years ago

@TockaAyman-VG Do you have a service defined?

You can check the Kubernetes Code Samples for more info [1]. I'll also add the information about the service to README.

[1] https://github.com/hazelcast/hazelcast-code-samples/blob/master/hazelcast-integration/kubernetes/hazelcast.yaml#L55

TockaAyman-VG commented 5 years ago

Yes I have a service for hazelcast which I took from the example you mentioned, here are all my yaml files rbac.yaml , config.yaml and hazelcast.yaml are all exactly like the example you mentioned except for changing the version to 3.8.2 to be like the one used by vertx app service.yaml

apiVersion: v1
kind: Service
metadata:
  name: appName
  labels:
    app: appName
spec:
  type: NodePort
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: appName

app deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: appName
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: appName
    spec:
      containers:
        - name: appName
          image: appImageURL:appImageTag
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /api/v1/health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5

and here is my ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: staging-ingress
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: appName
          servicePort: 8080
          path: "/api/v1/appName/*"

I am wondering if I should do some thing to direct my app to call the hazelcast service as I suspect that this line in logs is the problem INFO: No cluster-host specified so using address 10.16.0.12 the used ip address in this log statement is the Pod's IP address

leszko commented 5 years ago

@TockaAyman-VG ok, the other thing you may be missing is exposing the Hazelcast port 5701 from your PODs. Could you try to add the following and try again?

        - name: hazelcast
          containerPort: 5701
TockaAyman-VG commented 5 years ago

Unfortunately after adding it no thing changed, the same behavior happened

leszko commented 5 years ago

You probably need also a service which exposes the port 5701. Could you create one and try again?

  ports:
    - port: 8080
      protocol: TCP
TockaAyman-VG commented 5 years ago

Do you mean I need to expose 5701 on the nodes level? Or I need to have a specific pod and service to listen on port 5701? [By the way kubernetes does not allow me to expose node port outside of 30000-32767 range] I think hazelcast service itself is listening on 5701 but it is mapped to another node port by kubernetes However I exposed port 5701 from the app service and the same behavior happened Here is how kubernetes services are configured now 

> kubectl get service
NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                         AGE
hazelcast-service     LoadBalancer   10.19.251.152   130.211.196.234   5701:30931/TCP                  1h
kubernetes            ClusterIP      10.19.240.1     <none>            443/TCP                         1h
appservice            NodePort       10.19.245.224   <none>            8080:32295/TCP,5701:30368/TCP   50s

and this is the updated deployment.xml file 

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: appName
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: appName
    spec:
      containers:
        - name: appName
          image: AppImageURL:AppImageTag
          ports:
            - name: app
              containerPort: 8080
            - name: hazelcast
              containerPort: 5701
          livenessProbe:
            httpGet:
              path: /api/v1/health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5

and here is the updated app service.xml

apiVersion: v1
kind: Service
metadata:
  name: appName
  labels:
    app: appName
spec:
  type: NodePort
  ports:
    - name: app
      port: 8080
      protocol: TCP
    - name: hazelcast
      port: 5701
      protocol: TCP
  selector:
    app: appName
leszko commented 5 years ago

hazelcast-service should actually be enough. Could you paste here the YAML for hazecast-service?

leszko commented 5 years ago

Also, I don't see the matchSelector in your Deployment config. Could you try to add it?

  selector:
    matchLabels:
      app: appName
leszko commented 5 years ago

@TockaAyman-VG I've created also a Code Sample with the minimum configuration to run embedded Hazelcast on Kubernetes. You can have a look: https://github.com/leszko/hazelcast-code-samples/tree/kubernetes-embedded/hazelcast-integration/kubernetes/samples/embedded

Hope it'll help.

TockaAyman-VG commented 5 years ago

@leszko Finally, it worked.

I tried your code in a new cluster, it worked. Then I just changed from your image to mine with changing nothing else, it does not work. So I knew that the problem was inside my docker image.

There was two differences between yours and mine:

  1. You are configuring hazelcast through code while I am using xml file.
  2. You are using the hazelcast client directly while I am using it through vertx event bus client.

I noticed that vertx event bus client takes configuration for the internal hazelcast client through a file called cluster.xml as stated on their official documentation https://vertx.io/docs/vertx-hazelcast/java/ While hazelcast client takes configuration through hazelcast.xml file. So I renamed the file and every thing works just fine.

It is really confusing, but I am glade it worked.

Thank you very much, you saved my day.

leszko commented 5 years ago

Great to hear that @TockaAyman-VG . If you see any improvement that could be done to the configuration, could you please raise a related GH issue in the Hazelcast or Vertx GH repo? That could be helpful.

I'm closing the issue for now. Please feel free to reopen if needed.

vegegoku commented 4 years ago

I noticed that all samples are using same pod with replicas, which actually works and if i scale the pod i can see the other replicas joining the cluster, but my issue is that i have a two different pods and each is having its own cluster. @leszko what would you change in that example for 2 deployments to join the same cluster?!

leszko commented 4 years ago

@vegegoku You can create a service (can be just ClusterIP) and then point both PODs in the service selector. Then, use the service name in the Hazelcast Kubernetes configuration.