search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.22k stars 184 forks source link

[BUG]mongodb-4.0 pod role is none after restart #5912

Open ahjing99 opened 1 year ago

ahjing99 commented 1 year ago

➜ ~ kbcli version Kubernetes: v1.28.3-aliyun.1 KubeBlocks: 0.7.1-beta.4 kbcli: 0.7.1-beta.3

➜  ~ kbcli cluster describe pgstz-66788b8cdd -n kubeblocks-cloud-ns
Name: pgstz-66788b8cdd   Created Time: Nov 24,2023 11:13 UTC+0800
NAMESPACE             CLUSTER-DEFINITION   VERSION       STATUS     TERMINATION-POLICY
kubeblocks-cloud-ns   mongodb              mongodb-4.0   Updating   Delete

Endpoints:
COMPONENT   MODE        INTERNAL                                                               EXTERNAL
mongodb     ReadWrite   pgstz-66788b8cdd-mongodb.kubeblocks-cloud-ns.svc.cluster.local:27017   <none>

Topology:
COMPONENT   INSTANCE                     ROLE        STATUS    AZ                 NODE                                         CREATED-TIME
mongodb     pgstz-66788b8cdd-mongodb-0   <none>      Running   cn-zhangjiakou-a   cn-zhangjiakou.192.168.0.153/192.168.0.153   Nov 24,2023 11:17 UTC+0800
mongodb     pgstz-66788b8cdd-mongodb-1   primary     Running   cn-zhangjiakou-a   cn-zhangjiakou.192.168.0.153/192.168.0.153   Nov 24,2023 11:16 UTC+0800
mongodb     pgstz-66788b8cdd-mongodb-2   secondary   Running   cn-zhangjiakou-a   cn-zhangjiakou.192.168.0.153/192.168.0.153   Nov 24,2023 11:14 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mongodb     false       100m / 1             107374182400m / 1Gi     data:20Gi      kb-openebs-hostpath

Images:
COMPONENT   TYPE      IMAGE
mongodb     mongodb   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mongo:4.0

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION
kb-repo       Enabled       7 11 * * *        datafile        7d

Show cluster events: kbcli cluster list-events -n kubeblocks-cloud-ns pgstz-66788b8cdd

➜  ~ k describe cluster -n kubeblocks-cloud-ns pgstz-66788b8cdd
Name:         pgstz-66788b8cdd
Namespace:    kubeblocks-cloud-ns
Labels:       cloud.kubeblocks.io/cluster=pgstzw-new-cherry27
              cloud.kubeblocks.io/organization=free-demo
              cloud.kubeblocks.io/owner=245734566659112960
              clusterdefinition.kubeblocks.io/name=mongodb
              clusterversion.kubeblocks.io/name=mongodb-4.0
Annotations:  kubeblocks.io/ops-request: [{"name":"pgstz-66788b8cdd-restart-d2rxg","type":"Restart"}]
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-11-24T03:13:54Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:cloud.kubeblocks.io/cluster:
          f:cloud.kubeblocks.io/organization:
          f:cloud.kubeblocks.io/owner:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
          f:topologyKeys:
            .:
            v:"kubernetes.io/hostname":
        f:backup:
          .:
          f:cronExpression:
          f:enabled:
          f:method:
          f:pitrEnabled:
          f:repoName:
          f:retentionPeriod:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mongodb"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:volumeClaimTemplates:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
        f:terminationPolicy:
    Manager:      apiserver
    Operation:    Update
    Time:         2023-11-24T03:13:54Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/ops-request:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
    Manager:      manager
    Operation:    Update
    Time:         2023-11-24T03:16:47Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mongodb:
            .:
            f:consensusSetStatus:
              .:
              f:followers:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:membersStatus:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-11-24T03:17:01Z
  Resource Version:  10961726
  UID:               9af3a0a8-16ed-431b-8e08-16bbb8b5842c
Spec:
  Affinity:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
    Topology Keys:
      kubernetes.io/hostname
  Backup:
    Cron Expression:       7 11 * * *
    Enabled:               true
    Method:                datafile
    Pitr Enabled:          false
    Repo Name:             kb-repo
    Retention Period:      7d
  Cluster Definition Ref:  mongodb
  Cluster Version Ref:     mongodb-4.0
  Component Specs:
    Component Def Ref:  mongodb
    Monitor:            true
    Name:               mongodb
    No Create PDB:      false
    Replicas:           3
    Resources:
      Limits:
        Cpu:     1
        Memory:  1Gi
      Requests:
        Cpu:     100m
        Memory:  107374182400m
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  20Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  3
  Components:
    Mongodb:
      Consensus Set Status:
        Followers:
          Access Mode:  Readonly
          Name:         secondary
          Pod:          pgstz-66788b8cdd-mongodb-2
        Leader:
          Access Mode:  ReadWrite
          Name:         primary
          Pod:          pgstz-66788b8cdd-mongodb-1
      Members Status:
        Pod Name:  pgstz-66788b8cdd-mongodb-1
        Role:
          Access Mode:  ReadWrite
          Can Vote:     true
          Is Leader:    true
          Name:         primary
        Pod Name:       pgstz-66788b8cdd-mongodb-2
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         secondary
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-11-24T03:15:21Z
  Conditions:
    Last Transition Time:  2023-11-24T03:13:54Z
    Message:               The operator has started the provisioning of Cluster: pgstz-66788b8cdd
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-11-24T03:13:54Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-11-24T03:16:47Z
    Message:               pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-11-24T03:16:47Z
    Message:               pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Updating
Events:
  Type     Reason                    Age                From                Message
  ----     ------                    ----               ----                -------
  Normal   PreCheckSucceed           39m                cluster-controller  The operator has started the provisioning of Cluster: pgstz-66788b8cdd
  Normal   ApplyResourcesSucceed     39m                cluster-controller  Successfully applied for resources
  Normal   NeedWaiting               38m (x8 over 39m)  cluster-controller  waiting for restore "pgstz-66788b8cdd-mongodb-9af3a0a8-preparedata" successfully
  Normal   ComponentPhaseTransition  38m                cluster-controller  Create a new component
  Normal   ComponentPhaseTransition  37m                cluster-controller  component is Running
  Normal   AllReplicasReady          37m                cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              37m                cluster-controller  Cluster: pgstz-66788b8cdd is ready, current phase is Running
  Normal   Running                   37m                cluster-controller  Cluster: pgstz-66788b8cdd is ready, current phase is Running
  Normal   NeedWaiting               37m (x2 over 37m)  cluster-controller  waiting for restore "pgstz-66788b8cdd-mongodb-9af3a0a8-postready" successfully
  Normal   ComponentPhaseTransition  36m                cluster-controller  component is Updating
  Warning  ReplicasNotReady          36m                cluster-controller  pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        36m                cluster-controller  pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components

➜  ~ k logs pgstz-66788b8cdd-mongodb-0 -n kubeblocks-cloud-ns -c kb-checkrole >checkrole.txt
[checkrole.txt](https://github.com/apecloud/kubeblocks/files/13454996/checkrole.txt)
github-actions[bot] commented 11 months ago

This issue has been marked as stale because it has been open for 30 days with no activity