search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.02k stars 165 forks source link

[BUG]OB is always creating for Readiness probe failed on Azure #6700

Open ahjing99 opened 6 months ago

ahjing99 commented 6 months ago 
kbcli version
Kubernetes: v1.29.0
KubeBlocks: 0.7.2
kbcli: 0.7.2

apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: ocbase-eqzoyx
  namespace: default
  annotations:
    "kubeblocks.io/extra-env": '{"TENANT_NAME":"tenant3","ZONE_COUNT":"3","OB_CLUSTERS_COUNT":"1","TENANT_CPU":"2","TENANT_MEMORY":"2G","TENANT_DISK":"5G"}'
spec:
  clusterDefinitionRef: oceanbase
  clusterVersionRef: oceanbase-4.2.0.0-100010032023083021
  terminationPolicy: WipeOut
  componentSpecs:
    - name: ob-bundle
      componentDefRef: ob-bundle
      replicas: 3
      resources:
        requests:
          cpu: 3000m
          memory: 8Gi
        limits:
          cpu: 3000m
          memory: 8Gi
      volumeClaimTemplates:
        - name: data-file
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "50Gi"
        - name: data-log
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "50Gi"
        - name: log
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "10Gi"
        - name: workdir
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "100Mi"

      `kubectl apply -f test_create_ocbase-eqzoyx.yaml`

➜  ~ k describe pod ocbase-eqzoyx-ob-bundle-0
Name:         ocbase-eqzoyx-ob-bundle-0
Namespace:    default
Priority:     0
Node:         aks-userpool-38196425-vmss000002/10.224.1.184
Start Time:   Wed, 28 Feb 2024 10:35:40 +0800
Labels:       app.kubernetes.io/component=ob-bundle
              app.kubernetes.io/instance=ocbase-eqzoyx
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=oceanbase
              app.kubernetes.io/version=oceanbase-4.2.0.0-100010032023083021
              apps.kubeblocks.io/component-name=ob-bundle
              apps.kubeblocks.io/workload-type=Stateful
              apps.kubernetes.io/pod-index=0
              controller-revision-hash=ocbase-eqzoyx-ob-bundle-9b64b48b9
              statefulset.kubernetes.io/pod-name=ocbase-eqzoyx-ob-bundle-0
Annotations:  apps.kubeblocks.io/component-replicas: 3
Status:       Running
IP:           10.224.1.230
IPs:
  IP:           10.224.1.230
Controlled By:  StatefulSet/ocbase-eqzoyx-ob-bundle
Containers:
  observer-container:
    Container ID:  containerd://f0a8607c562f9c716dbcfbef8c7b494e846c4b807a3125ba3d86f13dc535305c
    Image:         docker.io/oceanbasedev/oceanbase-chart:4.2.0.0-100010032023083021
    Image ID:      docker.io/oceanbasedev/oceanbase-chart@sha256:bdb0feec2ce5ff0929285fd8991db96a22bf50a81d79100066aee4715be878e6
    Ports:         2881/TCP, 2882/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      bash
      -c
      /scripts/entrypoint.sh

    State:          Running
      Started:      Wed, 28 Feb 2024 10:36:01 +0800
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     3
      memory:  8Gi
    Requests:
      cpu:      3
      memory:   8Gi
    Readiness:  exec [cat /tmp/ready] delay=10s timeout=5s period=10s #success=1 #failure=10
    Environment Variables from:
      ocbase-eqzoyx-ob-bundle-env      ConfigMap  Optional: false
      ocbase-eqzoyx-ob-bundle-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               ocbase-eqzoyx-ob-bundle-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           ocbase-eqzoyx
      KB_COMP_NAME:              ob-bundle
      KB_CLUSTER_COMP_NAME:      ocbase-eqzoyx-ob-bundle
      KB_CLUSTER_UID_POSTFIX_8:  f5c545cb
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      OB_CLUSTERS_COUNT:         1
      TENANT_CPU:                2
      TENANT_DISK:               5G
      TENANT_MEMORY:             2G
      TENANT_NAME:               tenant3
      ZONE_COUNT:                3
      LD_LIBRARY_PATH:           /home/admin/oceanbase/lib
      CLUSTER_NAME:              $(KB_CLUSTER_COMP_NAME)
      POD_IP:                     (v1:status.podIP)
      DB_ROOT_PASSWORD:          <set to the key 'password' in secret 'ocbase-eqzoyx-conn-credential'>  Optional: false
      OB_HOME_DIR:               /home/admin/workdir
    Mounts:
      /home/admin/data-file from data-file (rw)
      /home/admin/data-log from data-log (rw)
      /home/admin/log from log (rw)
      /home/admin/workdir from workdir (rw)
      /kb-config from oceanbase-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s87nj (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  log:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  log-ocbase-eqzoyx-ob-bundle-0
    ReadOnly:   false
  workdir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  workdir-ocbase-eqzoyx-ob-bundle-0
    ReadOnly:   false
  data-file:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-file-ocbase-eqzoyx-ob-bundle-0
    ReadOnly:   false
  data-log:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-log-ocbase-eqzoyx-ob-bundle-0
    ReadOnly:   false
  oceanbase-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ocbase-eqzoyx-ob-bundle-oceanbase-config
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ocbase-eqzoyx-ob-bundle-oceanbase-scripts
    Optional:  false
  kube-api-access-s87nj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Normal   Scheduled               27m                    default-scheduler        Successfully assigned default/ocbase-eqzoyx-ob-bundle-0 to aks-userpool-38196425-vmss000002
  Normal   SuccessfulAttachVolume  27m                    attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-23466144-e7c9-4175-9677-019d63f0e2e3"
  Normal   SuccessfulAttachVolume  27m                    attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-5bf2cc98-8890-47a1-9955-3174572ed57b"
  Normal   SuccessfulAttachVolume  27m                    attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-5a0dcc43-77d5-43ad-983f-dff9a820d2d6"
  Normal   SuccessfulAttachVolume  27m                    attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-c4f2a230-083f-4003-8600-b5795a72c38c"
  Normal   Pulling                 27m                    kubelet                  Pulling image "docker.io/oceanbasedev/oceanbase-chart:4.2.0.0-100010032023083021"
  Normal   Pulled                  27m                    kubelet                  Successfully pulled image "docker.io/oceanbasedev/oceanbase-chart:4.2.0.0-100010032023083021" in 5.856s (5.856s including waiting)
  Normal   Created                 27m                    kubelet                  Created container observer-container
  Normal   Started                 27m                    kubelet                  Started container observer-container
  Warning  Unhealthy               2m31s (x168 over 27m)  kubelet                  Readiness probe failed: cat: /tmp/ready: No such file or directory

➜ ~ kbcli report cluster ocbase-eqzoyx --with-logs --all-containers reporting cluster information to report-cluster-ocbase-eqzoyx-2024-02-28-11-07-08.zip processing manifests OK processing events OK process pod logs OK ➜ ~ kbcli report kubeblocks --with-logs --all-containers --output yaml reporting KubeBlocks information to report-kubeblocks-2024-02-28-11-07-23.zip processing manifests OK processing events OK process pod logs OK report-kubeblocks-2024-02-28-11-07-23.zip report-cluster-ocbase-eqzoyx-2024-02-28-11-07-08.zip

github-actions[bot] commented 5 months ago

This issue has been marked as stale because it has been open for 30 days with no activity