[BUG]mysql Cluster Service can not connect

[ahjing99]

➜ ~ kbcli version Kubernetes: v1.27.3-gke.100 KubeBlocks: 0.7.0-alpha.13 kbcli: 0.7.0-alpha.13

      `kbcli cluster create  mysql-ndllsk             --termination-policy=WipeOut             --monitoring-interval=0 --enable-all-logs=false --cluster-definition=mysql --cluster-version=mysql-8.0.33 --set cpu=100m,memory=0.5Gi,replicas=1,storage=1Gi  --namespace default `

Cluster mysql-ndllsk created

➜  ~ kbcli cluster connect mysql-ndllsk
Connect to instance mysql-ndllsk-mysql-0
Defaulted container "mysql" out of: mysql, metrics, kb-checkrole, action-0, action-1, role-observe, role-agent-installer (init)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 563
Server version: 8.0.33 MySQL Community Server - GPL

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

➜  ~ kbcli cluster describe mysql-ndllsk
Name: mysql-ndllsk   Created Time: Sep 21,2023 16:45 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION        STATUS    TERMINATION-POLICY
default     mysql                mysql-8.0.33   Running   WipeOut

Endpoints:
COMPONENT   MODE        INTERNAL                                            EXTERNAL
mysql       ReadWrite   mysql-ndllsk-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE               ROLE      STATUS    AZ              NODE                                                CREATED-TIME
mysql       mysql-ndllsk-mysql-0   primary   Running   us-central1-b   gke-yjtest-default-pool-8a7e6e78-1jnn/10.128.0.27   Sep 21,2023 16:50 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE-SIZE   STORAGE-CLASS
mysql       false       200m / 200m          644245094400m / 644245094400m   data:1Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   docker.io/mysql:8.0.33

Show cluster events: kbcli cluster list-events -n default mysql-ndllsk
➜  ~

 `kbcli bench  sysbench benchtest-mysql-ndllsk --cluster mysql-ndllsk --host mysql-ndllsk-mysql --user root --password glpnn8gt --database benchtest  --namespace default --port 3306 --threads 4                     --type oltp_read_write --tables 4 --size 1000 --duration 10 `

Sysbench benchtest-mysql-ndllsk created

➜  ~ k logs benchtest-mysql-ndllsk-cleanup-knlww
run sysbench
check mysql database exists
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
mysql host mysql-ndllsk-mysql connect fail:(2003, "Can't connect to MySQL server on 'mysql-ndllsk-mysql' ([Errno 111] Connection refused)")
/usr/bin/sysbench
cd ./sysbench/src/lua;sysbench --db-driver=mysql --mysql-host=mysql-ndllsk-mysql --mysql-port=3306 --mysql-user=root --mysql-password=glpnn8gt --mysql-db=benchtest --table_size=1000 --tables=4 --time=10 --threads=4  oltp_read_write cleanup
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)

FATAL: unable to connect to MySQL server on host 'mysql-ndllsk-mysql', port 3306, aborting...
FATAL: error 2002: Can't connect to MySQL server on 'mysql-ndllsk-mysql' (115)
FATAL: `cleanup' function failed: ./oltp_common.lua:409: connection creation failed
Traceback (most recent call last):
  File "/datatestsuites/infratest.py", line 163, in <module>
    switchType(test_type)
  File "/datatestsuites/infratest.py", line 159, in switchType
    type_method()
  File "/datatestsuites/infratest.py", line 61, in Type_Sysbench
    run_sysbench(os_name, test_configs, test_flag)
  File "/datatestsuites/sysbench/run.py", line 118, in run_sysbench
    sysbench_run(cmd_head, db_host, db_port, db_user, db_password,
  File "/datatestsuites/sysbench/run.py", line 265, in sysbench_run
    raise Exception("sysbench cleanup error:" + str(ret_run))
Exception: sysbench cleanup error:1

➜  ~ k get pod mysql-ndllsk-mysql-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    apps.kubeblocks.io/component-replicas: "1"
    rs.apps.kubeblocks.io/primary: mysql-ndllsk-mysql-0
  creationTimestamp: "2023-09-21T08:50:19Z"
  generateName: mysql-ndllsk-mysql-
  labels:
    app.kubernetes.io/component: mysql
    app.kubernetes.io/instance: mysql-ndllsk
    app.kubernetes.io/managed-by: kubeblocks
    app.kubernetes.io/name: mysql
    app.kubernetes.io/version: mysql-8.0.33
    apps.kubeblocks.io/component-name: mysql
    apps.kubeblocks.io/workload-type: Replication
    controller-revision-hash: mysql-ndllsk-mysql-67c5ddb468
    kubeblocks.io/role: primary
    statefulset.kubernetes.io/pod-name: mysql-ndllsk-mysql-0
  name: mysql-ndllsk-mysql-0
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: mysql-ndllsk-mysql
    uid: 89ba97da-dc3e-4e8b-9be9-29e0b43057ca
  resourceVersion: "96072"
  uid: e5123715-5fb2-49e7-a0c4-5833022ab1c4
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: kb-data
            operator: In
            values:
            - "true"
        weight: 100
    podAntiAffinity: {}
  containers:
  - command:
    - bash
    - -c
    - docker-entrypoint.sh mysqld --server-id $(( ${KB_POD_NAME##*-} + 1))
    env:
    - name: KB_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: KB_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: KB_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: KB_SA_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: KB_NODENAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: KB_HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KB_POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: KB_POD_IPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: KB_HOSTIP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KB_PODIP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: KB_PODIPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: KB_CLUSTER_NAME
      value: mysql-ndllsk
    - name: KB_COMP_NAME
      value: mysql
    - name: KB_CLUSTER_COMP_NAME
      value: mysql-ndllsk-mysql
    - name: KB_CLUSTER_UID_POSTFIX_8
      value: c688fc4e
    - name: KB_POD_FQDN
      value: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    - name: MYSQL_ROOT_HOST
      value: '%'
    - name: MYSQL_ROOT_USER
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
          optional: false
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
          optional: false
    envFrom:
    - configMapRef:
        name: mysql-ndllsk-mysql-env
    - configMapRef:
        name: mysql-ndllsk-mysql-rsm-env
        optional: false
    image: docker.io/mysql:8.0.33
    imagePullPolicy: IfNotPresent
    name: mysql
    ports:
    - containerPort: 3306
      name: mysql
      protocol: TCP
    resources:
      limits:
        cpu: 200m
        memory: 644245094400m
      requests:
        cpu: 200m
        memory: 644245094400m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: data
    - mountPath: /etc/mysql/conf.d
      name: mysql-config
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  - command:
    - /scripts/agamotto.sh
    env:
    - name: KB_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: KB_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: KB_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: KB_SA_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: KB_NODENAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: KB_HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KB_POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: KB_POD_IPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: KB_HOSTIP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KB_PODIP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: KB_PODIPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: KB_CLUSTER_NAME
      value: mysql-ndllsk
    - name: KB_COMP_NAME
      value: mysql
    - name: KB_CLUSTER_COMP_NAME
      value: mysql-ndllsk-mysql
    - name: KB_CLUSTER_UID_POSTFIX_8
      value: c688fc4e
    - name: KB_POD_FQDN
      value: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    - name: DB_TYPE
      value: MySQL
    - name: ENDPOINT
      value: localhost:3306
    - name: MYSQL_USER
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
          optional: false
    - name: MYSQL_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
          optional: false
    envFrom:
    - configMapRef:
        name: mysql-ndllsk-mysql-env
    - configMapRef:
        name: mysql-ndllsk-mysql-rsm-env
        optional: false
    image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    imagePullPolicy: IfNotPresent
    name: metrics
    ports:
    - containerPort: 9104
      name: http-metrics
      protocol: TCP
    resources:
      limits:
        cpu: "0"
        memory: "0"
      requests:
        cpu: "0"
        memory: "0"
    securityContext:
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /opt/agamotto
      name: agamotto-configuration
    - mountPath: /data/mysql
      name: data
    - mountPath: /var/log/kubeblocks
      name: log-data
    - mountPath: /scripts
      name: scripts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  - command:
    - probe
    - --port
    - "3501"
    env:
    - name: KB_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: KB_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: KB_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: KB_SA_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: KB_NODENAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: KB_HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KB_POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: KB_POD_IPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: KB_HOSTIP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KB_PODIP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: KB_PODIPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: KB_CLUSTER_NAME
      value: mysql-ndllsk
    - name: KB_COMP_NAME
      value: mysql
    - name: KB_CLUSTER_COMP_NAME
      value: mysql-ndllsk-mysql
    - name: KB_CLUSTER_UID_POSTFIX_8
      value: c688fc4e
    - name: KB_POD_FQDN
      value: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    - name: KB_SERVICE_USER
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
    - name: KB_SERVICE_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
    - name: KB_SERVICE_PORT
      value: "3306"
    - name: KB_DATA_PATH
      value: /var/lib/mysql
    - name: KB_SERVICE_ROLES
      value: '{}'
    - name: KB_SERVICE_CHARACTER_TYPE
      value: mysql
    - name: KB_WORKLOAD_TYPE
      value: Replication
    - name: KB_RSM_ACTION_SVC_LIST
    envFrom:
    - configMapRef:
        name: mysql-ndllsk-mysql-env
    - configMapRef:
        name: mysql-ndllsk-mysql-rsm-env
        optional: false
    image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-alpha.13
    imagePullPolicy: IfNotPresent
    name: kb-checkrole
    ports:
    - containerPort: 3501
      name: probe-http-port
      protocol: TCP
    - containerPort: 50001
      name: probe-grpc-port
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /v1.0/bindings/mysql?operation=checkRole&workloadType=Replication
        port: 3501
        scheme: HTTP
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "0"
        memory: "0"
      requests:
        cpu: "0"
        memory: "0"
    startupProbe:
      failureThreshold: 3
      periodSeconds: 10
      successThreshold: 1
      tcpSocket:
        port: 3501
      timeoutSeconds: 1
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: data
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  - command:
    - /role-probe/agent
    - -port
    - "36501"
    - -export-all-vars
    - -form
    - /role
    - curl http://127.0.0.1:3501/v1.0/bindings/mysql?operation=checkRole&workloadType=Replication
    env:
    - name: KB_RSM_USERNAME
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
    - name: KB_RSM_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
    image: free6om/kubeblocks:latest
    imagePullPolicy: IfNotPresent
    name: action-0
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /role-probe
      name: role-agent
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  - command:
    - /role-probe/agent
    - -port
    - "36502"
    - -export-all-vars
    - -form
    - /role
    - echo $v_KB_RSM_LAST_STDOUT | jq -r '.role' | tr '[:upper:]' '[:lower:]' | xargs
      echo -n
    env:
    - name: KB_RSM_USERNAME
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
    - name: KB_RSM_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
    image: jetbrainsinfra/jq:latest
    imagePullPolicy: IfNotPresent
    name: action-1
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /role-probe
      name: role-agent
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  - command:
    - role-agent
    - --port
    - "7373"
    env:
    - name: KB_RSM_USERNAME
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
    - name: KB_RSM_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
    - name: KB_RSM_ACTION_SVC_LIST
      value: '[36501,36502]'
    - name: KB_SERVICE_USER
      valueFrom:
        secretKeyRef:
          key: username
          name: mysql-ndllsk-conn-credential
    - name: KB_SERVICE_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-ndllsk-conn-credential
    - name: KB_RSM_SERVICE_PORT
      value: "3306"
    - name: KB_SERVICE_PORT
      value: "3306"
    image: apecloud/kubeblocks-role-agent:latest
    imagePullPolicy: IfNotPresent
    name: role-observe
    ports:
    - containerPort: 7373
      name: role-observe
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /bin/grpc_health_probe
        - -addr=localhost:7373
      failureThreshold: 2
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: mysql-ndllsk-mysql-0
  initContainers:
  - command:
    - cp
    - /app/shell2http
    - /role-probe/agent
    image: msoap/shell2http:1.16.0
    imagePullPolicy: IfNotPresent
    name: role-agent-installer
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /role-probe
      name: role-agent
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzcpj
      readOnly: true
  nodeName: gke-yjtest-default-pool-8a7e6e78-1jnn
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: kb-mysql-ndllsk
  serviceAccountName: kb-mysql-ndllsk
  subdomain: mysql-ndllsk-mysql-headless
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: kb-data
    operator: Equal
    value: "true"
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-mysql-ndllsk-mysql-0
  - configMap:
      defaultMode: 292
      name: mysql-ndllsk-mysql-agamotto-configuration
    name: agamotto-configuration
  - configMap:
      defaultMode: 420
      name: mysql-ndllsk-mysql-mysql-replication-config
    name: mysql-config
  - configMap:
      defaultMode: 365
      name: mysql-ndllsk-mysql-mysql-scripts
    name: scripts
  - emptyDir: {}
    name: log-data
  - emptyDir: {}
    name: role-agent
  - name: kube-api-access-nzcpj
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-09-21T08:50:26Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-09-21T08:50:36Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-09-21T08:50:36Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-09-21T08:50:19Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://26de676438eb67bce7f90d480f62c0aca9b1b878898546e8c94d204353b50366
    image: docker.io/free6om/kubeblocks:latest
    imageID: docker.io/free6om/kubeblocks@sha256:66e343105e8d6481d908b59935a8e4a7e22b0524e8e3cff948207e2567134bcb
    lastState: {}
    name: action-0
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-09-21T08:50:26Z"
  - containerID: containerd://9c93c634db2a9dc0c3ba5003d9697a5718f82f2d96e0c01d70d84b769302c96a
    image: docker.io/jetbrainsinfra/jq:latest
    imageID: docker.io/jetbrainsinfra/jq@sha256:bf917fad410a1f44759400299e0557513c7f1f51ca00e719c6607a81f2e5308d
    lastState: {}
    name: action-1
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-09-21T08:50:26Z"
  - containerID: containerd://fd2b943e3974125982ddca654179a22da04b751a48f0818cab8402cbea3a27aa
    image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-alpha.13
    imageID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:b6925598198d8dcbc86e8892f8096fea16f77e50c93ea7407b4861d74a6249b5
    lastState: {}
    name: kb-checkrole
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-09-21T08:50:26Z"
  - containerID: containerd://7fb467c3ded464ba7a9ea572c0dde997364d89c0242a62e514ff0d8b63d690d6
    image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    imageID: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
    lastState: {}
    name: metrics
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-09-21T08:50:26Z"
  - containerID: containerd://7beaa6808c4d75e770267d74616fe65484401da0ebb75522ee870be087bb44d3
    image: docker.io/library/mysql:8.0.33
    imageID: docker.io/library/mysql@sha256:ea68e51ffe9b96fef6076f1218af11301aeaf13c6201e0ec9aaef5791d5ddc5d
    lastState: {}
    name: mysql
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-09-21T08:50:26Z"
  - containerID: containerd://4b3411b205c45b935e9b6325bf7177c45ed343daddabbf48e0964386618c5f95
    image: docker.io/apecloud/kubeblocks-role-agent:latest
    imageID: docker.io/apecloud/kubeblocks-role-agent@sha256:094c90431b37fbdae13a85b491628fb05394f00de423a5686141ec63867181c2
    lastState: {}
    name: role-observe
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-09-21T08:50:26Z"
  hostIP: 10.128.0.27
  initContainerStatuses:
  - containerID: containerd://205ae3ae025b957a27b3ab81c5d27a1b9612292a515964f8422242bffe43a679
    image: docker.io/msoap/shell2http:1.16.0
    imageID: docker.io/msoap/shell2http@sha256:a20bdde2f679de2cba6bf3d9f470489c7836d4d0d28232a2b295450809cd43ef
    lastState: {}
    name: role-agent-installer
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://205ae3ae025b957a27b3ab81c5d27a1b9612292a515964f8422242bffe43a679
        exitCode: 0
        finishedAt: "2023-09-21T08:50:25Z"
        reason: Completed
        startedAt: "2023-09-21T08:50:25Z"
  phase: Running
  podIP: 10.20.0.25
  podIPs:
  - ip: 10.20.0.25
  qosClass: Burstable
  startTime: "2023-09-21T08:50:19Z"

➜  ~ k get svc
NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                AGE
kubebench-exporter            ClusterIP   None          <none>        9187/TCP                               66m
kubernetes                    ClusterIP   10.24.0.1     <none>        443/TCP                                179m
mysql-ndllsk-mysql            ClusterIP   10.24.15.91   <none>        3306/TCP                               40m
mysql-ndllsk-mysql-headless   ClusterIP   None          <none>        3306/TCP,9104/TCP,3501/TCP,50001/TCP   40m

[iziang]

It's not a network issue, the service mysql-ndllsk-mysql doesn't match with any endpoints. Please check the service selector and the health status of pods.

kg svc
NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                AGE
kubebench-exporter            ClusterIP   None          <none>        9187/TCP                               114m
kubernetes                    ClusterIP   10.24.0.1     <none>        443/TCP                                3h47m
mysql-ndllsk-mysql            ClusterIP   10.24.15.91   <none>        3306/TCP                               87m
mysql-ndllsk-mysql-headless   ClusterIP   None          <none>        3306/TCP,9104/TCP,3501/TCP,50001/TCP   87m
mysql-tlyubr-mysql            ClusterIP   10.24.9.30    <none>        3306/TCP                               39m
mysql-tlyubr-mysql-headless   ClusterIP   None          <none>        3306/TCP,9104/TCP,3501/TCP,50001/TCP   39m

kg ep
NAME                          ENDPOINTS                                                      AGE
kubebench-exporter            10.20.0.28:9187,10.20.1.16:9187,10.20.2.15:9187                115m
kubernetes                    10.128.0.24:443                                                3h48m
mysql-ndllsk-mysql            <none>                                                         88m
mysql-ndllsk-mysql-headless   10.20.0.25:3501,10.20.0.25:3306,10.20.0.25:50001 + 1 more...   88m
mysql-tlyubr-mysql            <none>                                                         40m
mysql-tlyubr-mysql-headless   10.20.1.20:3501,10.20.2.14:3501,10.20.1.20:3306 + 5 more...    40m