Issue with Kubernetes embedded configuration not picking up environment variables

Sreini commented 3 years ago

I am having issues with the hazelcast k8s plugin not picking up the correct configuration.

In my kubernetes deployment I have defined the following environment variables:

        - name: HZ_NETWORK_JOIN_KUBERNETES_SERVICENAME
          value: my-service-name
        - name: HZ_NETWORK_JOIN_MULTICAST_ENABLED
          value: 'false'
        - name: HZ_NETWORK_JOIN_KUBERNETES_ENABLED
          value: 'true'

In accordance with the guidelines for setting up embedded hazelcast in kubernetes according to this repo and in combination with the official documentation: https://docs.hazelcast.com/imdg/latest/configuration/overriding-configuration-settings.html. However the Service Name does not seem to be picked up. When I start my application I get the following logging:

14:37:08.698 [main] INFO  c.h.s.d.integration.DiscoveryService - [10.7.1.206]:5701 [dev] [4.2] Auto-detection selected discovery strategy: class com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategyFactory
14:37:08.705 [main] INFO  c.h.s.d.integration.DiscoveryService - [10.7.1.206]:5701 [dev] [4.2] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-port: 0, service-label: null, service-label-value: true, namespace: my-namespace, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, use-node-name-as-external-address: false, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
14:37:08.711 [main] INFO  c.h.s.d.integration.DiscoveryService - [10.7.1.206]:5701 [dev] [4.2] Kubernetes Discovery activated with mode: KUBERNETES_API

Specifically, it says right there: service-name: null, whereas according to my configuration it should be my-service-name

Sreini commented 3 years ago

I get the same message if I try using the Java configuration:

config.getNetworkConfig().getJoin().getKubernetesConfig().setProperty("service-name", "my-service-name");

alparslanavci commented 3 years ago

@Sreini thanks for reporting the issue. I tried with your configuration using the following deployment and it works as expected:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hazelcast-cluster
spec:
  serviceName: hazelcast-cluster
  replicas: 3
  selector:
    matchLabels:
      app: hazelcast-cluster
  template:
    metadata:
      labels:
        app: hazelcast-cluster
    spec:
      containers:
        - name: hazelcast-cluster
          image: hazelcast/hazelcast:4.2
          ports:
          - containerPort: 5701
          env:
          - name: HZ_NETWORK_JOIN_KUBERNETES_SERVICENAME
            value: hazelcast-cluster
          - name: HZ_NETWORK_JOIN_MULTICAST_ENABLED
            value: 'false'
          - name: HZ_NETWORK_JOIN_KUBERNETES_ENABLED
            value: 'true'

---

apiVersion: v1
kind: Service
metadata:
  name: hazelcast-cluster
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: hazelcast-cluster
  ports:
    - name: tcp-hazelcast
      port: 5701

2021-04-14 09:46:23,605 [ INFO] [main] [c.h.system]: [10.0.3.27]:5701 [dev] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [10.0.3.27]:5701
2021-04-14 09:46:24,372 [ INFO] [main] [c.h.s.d.i.DiscoveryService]: [10.0.3.27]:5701 [dev] [4.2] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: hazelcast-cluster, service-port: 0, service-label: null, service-label-value: true, namespace: default, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, use-node-name-as-external-address: false, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2021-04-14 09:46:24,377 [ INFO] [main] [c.h.s.d.i.DiscoveryService]: [10.0.3.27]:5701 [dev] [4.2] Kubernetes Discovery activated with mode: KUBERNETES_API

Could you please check if the env variables are set correctly in your deployment? You can use the following command to see them:

kubectl exec [POD_NAME] -- printenv

Sreini commented 3 years ago

I checked the environment variables as you described:

HZ_NETWORK_JOIN_KUBERNETES_SERVICENAME=my-service-name HZ_NETWORK_JOIN_MULTICAST_ENABLED=false HZ_NETWORK_JOIN_KUBERNETES_ENABLED=true

however I am still getting in the logs. Is there anything else I might be doing wrong?

alparslanavci commented 3 years ago

Could you please share your deployment setup with us? Are there any chances in the container image or the configuration?

ndrgrbr commented 9 months ago

It depends how your config is loaded. To use env vars must be something like that: https://javadoc.io/static/com.hazelcast/hazelcast/4.2.8/com/hazelcast/config/Config.html#load--

It tries to load Hazelcast configuration from a list of well-known locations, and then applies overrides found in environment variables/system properties When no location contains Hazelcast configuration then it returns default.

Note that the same mechanism is used when calling Hazelcast.newHazelcastInstance()

hazelcast / hazelcast-kubernetes

Issue with Kubernetes embedded configuration not picking up environment variables #306