[bitnami/mariadb-galera] Expose Galera (headless) service outside Kubernetes Cluster

[amit-k-yadav]

Which chart:

• mariadb-galera (version - 5.3.4)

Bug description & steps to re-produce

• I am trying to deploy a Galera cluster on multiple Kubernetes clusters (one Galera node per Kubernetes cluster) across multiple GCP regions for high availability during regional failures in Google Cloud Platform.

• I have set the values of replicaCount (here) to 1 in order to have just one node to start with.

• Now since headless services are not reachable outside Kubernetes clusters, I have added an additional service specifically for this only running pod. Below is the service definition:

  apiVersion: v1
  kind: Service
  metadata:
  name: {{ printf "%s-external" (include "common.names.fullname" .) }}
  labels: {{- include "common.labels.standard" . | nindent 4 }}
  {{- if .Values.service.headless.annotations }}
  annotations: {{- include "common.tplvalues.render" (dict "value" .Values.service.headless.annotations "context" $) | nindent 4 }}
  {{- end }}
  spec:
  type: LoadBalancer               # Modified
  externalTrafficPolicy: Local   # Modified
  ports:
    - name: galera
      port: 4567
      targetPort: galera
    - name: ist
      port: 4568
      targetPort: ist
    - name: sst
      port: 4444
      targetPort: sst
  selector: {{- include "common.labels.matchLabels" . | nindent 4 }}

• I took the already existing headless service and created new file (external-galera-svc.yaml) with the above content. When I install this chart.

• I have also changed the statefulset.yaml file here to use the LoadBalancer (external service) instead of the headless service for MARIADB_GALERA_CLUSTER_ADDRESS. Below are the changes:

  # Replace headless with external
  - name: MARIADB_GALERA_CLUSTER_ADDRESS
  value: "gcomm://{{ template "common.names.fullname" . }}-external.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}"

• I am getting the below error:

  ...
  
  2021-01-05  6:39:11 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
  2021-01-05  6:39:11 0 [Note] WSREP: Passing config to GCS: base_dir = /bitnami/mariadb/data/; base_host = 10.48.2.60; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug
  = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /bitnami/mariadb/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc
  2021-01-05  6:39:11 0 [Note] WSREP: Start replication
  2021-01-05  6:39:11 0 [Note] WSREP: Connecting with bootstrap option: 0
  2021-01-05  6:39:11 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
  2021-01-05  6:39:11 0 [Note] WSREP: protonet asio version 0
  2021-01-05  6:39:11 0 [Note] WSREP: Using CRC-32C for message checksums.
  2021-01-05  6:39:11 0 [Note] WSREP: backend: asio
  2021-01-05  6:39:11 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
  2021-01-05  6:39:11 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory)
  2021-01-05  6:39:11 0 [Note] WSREP: restore pc from disk failed
  2021-01-05  6:39:11 0 [Note] WSREP: GMCast version 0
  2021-01-05  6:39:11 0 [Note] WSREP: (b891cdb1-a67b, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
  2021-01-05  6:39:11 0 [Note] WSREP: (b891cdb1-a67b, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
  2021-01-05  6:39:11 0 [Note] WSREP: EVS version 1
  2021-01-05  6:39:11 0 [Note] WSREP: gcomm: connecting to group 'galera', peer 'external-mariadb-galera-external.galera.svc.cluster.local:'
  2021-01-05  6:39:14 0 [Note] WSREP: EVS version upgrade 0 -> 1
  2021-01-05  6:39:14 0 [Note] WSREP: PC protocol upgrade 0 -> 1
  2021-01-05  6:39:14 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
  2021-01-05  6:39:14 0 [Note] WSREP: view(view_id(NON_PRIM,b891cdb1-a67b,1) memb {
        b891cdb1-a67b,0
  } joined {
  } left {
  } partitioned {
  })
  2021-01-05  6:39:15 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50132S), skipping check
  2021-01-05  6:39:44 0 [Note] WSREP: PC protocol downgrade 1 -> 0
  2021-01-05  6:39:44 0 [Note] WSREP: view((empty))
  2021-01-05  6:39:44 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
        at gcomm/src/pc.cpp:connect():160
  2021-01-05  6:39:44 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out)
  2021-01-05  6:39:44 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1632: Failed to open channel 'galera' at 'gcomm://external-mariadb-galera-external.galera.svc.cluster.local': -110 (Connection timed out)
  2021-01-05  6:39:44 0 [ERROR] WSREP: gcs connect failed: Connection timed out
  2021-01-05  6:39:44 0 [ERROR] WSREP: wsrep::connect(gcomm://external-mariadb-galera-external.galera.svc.cluster.local) failed: 7
  2021-01-05  6:39:44 0 [ERROR] Aborting

Expected behavior

• Since there is only one pod running, this service should redirect the traffic to it and the cluster should start running with one node.
• I have tried to keep the headless service with replicaCount: 1 and it works fine, it should work with LoadBalancer service too.

Version of Helm and Kubernetes:

• Output of helm version:

  version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

• Output of kubectl version:

  Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.13-gke.2001", GitCommit:"00c919adfe4adf308bcd7c02838f2a1b60482f02", GitTreeState:"clean", BuildDate:"2020-11-06T18:24:02Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

[andresbono]

Hi @amit-k-yadav, I'm afraid that what you are trying to achieve is not supported by this Helm chart. MariaDB Galera needs to reach as many peers as possible, if not, you will be loosing the high availability capabilities. Note that the headless service is binded to all the galera peers while your external service is going to be binded to only one of them.

[amit-k-yadav]

Thank you @andresbono for your reply. I understand what you are saying and that is the reason why I have reduced the replicaCount to 1 for now. This way, there is only one peer for now. This worked perfectly fine with Headless service.

I am planning to have 1 LoadBalancer service per pod (i.e peer) and mention all the services in MARIADB_GALERA_CLUSTER_ADDRESS (i.e. gcomm://...) . This however, will be the next step, for now it should work right since there is only one pod and the service is dedicated to that pod particularly.

I am not sure why Headless service works and LoadBalancer failes for one peer.

[andresbono]

After some investigation, I couldn't make it work yet. It seems the problem is related to wsrep_node_address. This value should be the same than one of the used in the wsrep_cluster_address.

The container guesses the value for wsrep_node_address by executing hostname -i, which returns the pod IP. link.

When a headless service is used with selectors, it points directly to the pod IP, so wsrep_node_address and wsrep_cluster_address are compatible but that's not the case when you define the service type as LoadBalancer...

To workaround this issue I tried to use the MARIADB_GALERA_NODE_ADDRESS env-var. You will need to modify the statefulset.yaml file in order to add it (or alternatively, you can use extraEnvVars):

- name: MARIADB_GALERA_CLUSTER_ADDRESS
  value: "gcomm://{{ template "common.names.fullname" . }}-external.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}"
- name: MARIADB_GALERA_NODE_ADDRESS
  value: "{{ template "common.names.fullname" . }}-external.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}"

But even after setting it, the error is still the same. I'm sharing this information with you so that you can continue debugging it on your side.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[leokhachatorians]

I'm also interested in this issue. Specifically cross-regional setup across multiple clusters.

[miguelaeh]

Hi @leokhachatorians , As @andresbono said it seems the target configuration of this issue is not supported by the helm Chart at this moment https://github.com/bitnami/charts/issues/4887#issuecomment-754590126. You could try a configuration like the one explained here https://github.com/bitnami/charts/issues/4887#issuecomment-754605180 but it seems he is also having issue with that

[leokhachatorians]

@miguelaeh Ah understood. I ended up more or less doing a bit of a hack to the libmariadbgalera.sh to adjust the manner in which it would determine when to bootstrap the cluster; if anyone ends up going down a similar path.

[miguelaeh]

thank you for sharing it @leokhachatorians !

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.