[bitnami/mariadb-galera] [question]Does anyone use mariadb-galera in istio?

[lcfang]

Which chart:

apiVersion: v1
name: mariadb-galera
version: 4.0.0
appVersion: 10.5.4

Does anyone use mariadb-galera in istio have problem as below?

[root@node-1]# kubectl  get po -ntest-istio -owide
NAME                           READY   STATUS             RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
istio-mysql-mariadb-galera-0   2/2     Running            0          21h   10.233.65.94    node-2   <none>           <none>
istio-mysql-mariadb-galera-1   1/2     CrashLoopBackOff   227        21h   10.233.64.136   node-1   <none>           <none>

After pod0 run normally，pod1 can't be running status. when I get the log of pod1, I have this :

2020-08-24 12:07:59 0 [Note] WSREP: gcomm: connecting to group 'galera', peer 'istio-mysql-mariadb-galera-headless.test-istio.svc.cluster.local:'
2020-08-24 12:08:02 0 [Note] WSREP: EVS version upgrade 0 -> 1
2020-08-24 12:08:02 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2020-08-24 12:08:02 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2020-08-24 12:08:02 0 [Note] WSREP: view(view_id(NON_PRIM,73e901e5-95ba,1) memb {
        73e901e5-95ba,0
} joined {
} left {
} partitioned {
})
2020-08-24 12:08:03 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50146S), skipping check
2020-08-24 12:08:08 0 [Note] WSREP: (73e901e5-95ba, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.233.65.94:4567 timed out, no messages seen in PT3S, socket stats: rtt: 36 rttvar: 18 rto: 201000 lost: 0 last_data_recv: 3001 cwnd: 10 last_queued_since: 3000019431 last_delivered_since: 3000019431 send_queue_length: 0 send_queue_bytes: 0
2020-08-24 12:08:12 0 [Note] WSREP: (73e901e5-95ba, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.233.65.94:4567 timed out, no messages seen in PT3S, socket stats: rtt: 32 rttvar: 16 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 3000047622 last_delivered_since: 3000047622 send_queue_length: 0 send_queue_bytes: 0
2020-08-24 12:08:16 0 [Note] WSREP: (73e901e5-95ba, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.233.65.94:4567 timed out, no messages seen in PT3S, socket stats: rtt: 45 rttvar: 22 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 3000037586 last_delivered_since: 3000037586 send_queue_length: 0 send_queue_bytes: 0
2020-08-24 12:08:20 0 [Note] WSREP: (73e901e5-95ba, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.233.65.94:4567 timed out, no messages seen in PT3S, socket stats: rtt: 43 rttvar: 21 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999993956 last_delivered_since: 2999993956 send_queue_length: 0 send_queue_bytes: 0
2020-08-24 12:08:25 0 [Note] WSREP: (73e901e5-95ba, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.233.65.94:4567 timed out, no messages seen in PT3S, socket stats: rtt: 48 rttvar: 24 rto: 200000 lost: 0 last_data_recv: 3500 cwnd: 10 last_queued_since: 3499986111 last_delivered_since: 3499986111 send_queue_length: 0 send_queue_bytes: 0
2020-08-24 12:08:30 0 [Note] WSREP: (73e901e5-95ba, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.233.65.94:4567 timed out, no messages seen in PT3S, socket stats: rtt: 44 rttvar: 22 rto: 200000 lost: 0 last_data_recv: 3001 cwnd: 10 last_queued_since: 3000018899 last_delivered_since: 3000018899 send_queue_length: 0 send_queue_bytes: 0
2020-08-24 12:08:32 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2020-08-24 12:08:32 0 [Note] WSREP: view((empty))
2020-08-24 12:08:32 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():160
2020-08-24 12:08:32 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out)
2020-08-24 12:08:32 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1632: Failed to open channel 'galera' at 'gcomm://istio-mysql-mariadb-galera-headless.test-istio.svc.cluster.local': -110 (Connection timed out)
2020-08-24 12:08:32 0 [ERROR] WSREP: gcs connect failed: Connection timed out
2020-08-24 12:08:32 0 [ERROR] WSREP: wsrep::connect(gcomm://istio-mysql-mariadb-galera-headless.test-istio.svc.cluster.local) failed: 7
2020-08-24 12:08:32 0 [ERROR] Aborting
Warning: Memory not freed: 48

Additional context

[root@node-1 ~]# istioctl version
client version: 1.7.0-rc.1
control plane version: 1.7.0-rc.1
data plane version: 1.7.0-rc.1 (5 proxies)
[root@node-1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

In the above environment, I can install mariadb-galera success without istio, I am not sure the reason is istio or not. Hope someone to discuss this.

[lcfang]

Few days ago, I can install success in istio but with wrong data synchronism, but now I have this failed log to start, It is so strange.

[andresbono]

Hi @lcfang, can you share with us the values that you are using to deploy the chart? We would like to test them deploying the chart in a kubernetes cluster not running istio.

[lcfang]

The command installing mariadb-galera is:

helm install --set image.debug=true istio-mysql  ./mariadb-galera --set global.storageClass=longhorn --namespace test-istio --set serviceAccount.create=true --set rbac.create=true --set rootUser.password=lcftest  --set securityContext.fsGroup=0 --set securityContext.runAsUser=0

As you see, I use longhorn which also can be installed by helm my storage, but it is not the requirement.

BTW, I can install successfully without istio in the same environment, but failed with istio.

[andresbono]

BTW, I can install successfully without istio in the same environment, but failed with istio.

Thanks, that is what we wanted to verify. If that is the case, you might want to review the requirements of Istio 1.7: https://istio.io/latest/docs/ops/deployment/requirements/

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[uluzox]

Adding

podAnnotations: {
  sidecar.istio.io/inject: "false"
}

to the values.yaml or the values-production.yaml disables istio sidecar injection for this workload only and makes the galera cluster work again.

[andresbono]

Thanks for the tip, @uluzox! 👍