search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.14k stars 176 forks source link

[BUG]openldap VolumeExpansion ops is always Running #5228

Closed ahjing99 closed 1 year ago

ahjing99 commented 1 year ago

➜ ~ kbcli version Kubernetes: v1.27.3-gke.100 KubeBlocks: 0.7.0-alpha.15 kbcli: 0.7.0-alpha.15


      `helm repo add kubeblocks  https://jihulab.com/api/v4/projects/150246/packages/helm/stable`

"kubeblocks" already exists with the same configuration, skipping

      `helm upgrade --install openldap kubeblocks/openldap --version 0.7.0-alpha.15 `

Release "openldap" does not exist. Installing it now.
NAME: openldap
LAST DEPLOYED: Fri Sep 22 13:39:01 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1

    `kbcli cluster create  oldap-yydccx             --termination-policy=DoNotTerminate             --monitoring-interval=0 --enable-all-logs=false --cluster-definition=openldap --cluster-version=openldap-2.4.57 --set cpu=100m,memory=0.5Gi,replicas=2,storage=1Gi  --namespace default `

Cluster oldap-yydccx created

   `kbcli cluster describe oldap-yydccx --namespace default `

Name: oldap-yydccx   Created Time: Sep 22,2023 13:39 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     openldap             openldap-2.4.57   Running   DoNotTerminate

Endpoints:
COMPONENT          MODE        INTERNAL                                                      EXTERNAL
openldap-compdef   ReadWrite   oldap-yydccx-openldap-compdef.default.svc.cluster.local:389   <none>

Topology:
COMPONENT          INSTANCE                          ROLE     STATUS    AZ              NODE                                                   CREATED-TIME
openldap-compdef   oldap-yydccx-openldap-compdef-0   <none>   Running   us-central1-f   gke-cluster-1-default-pool-55dff4bd-xnbj/10.128.0.35   Sep 22,2023 13:39 UTC+0800
openldap-compdef   oldap-yydccx-openldap-compdef-1   <none>   Running   us-central1-f   gke-cluster-1-default-pool-55dff4bd-pvwd/10.128.0.34   Sep 22,2023 13:39 UTC+0800

Resources Allocation:
COMPONENT          DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
openldap-compdef   false       100m / 100m          512Mi / 512Mi           data:1Gi       kb-default-sc

Images:
COMPONENT          TYPE               IMAGE
openldap-compdef   openldap-compdef   docker.io/osixia/openldap:1.5.0

Show cluster events: kbcli cluster list-events -n default oldap-yydccx

 `kbcli cluster volume-expand oldap-yydccx --auto-approve                 --components openldap-compdef                 --volume-claim-templates data                 --storage 6Gi --namespace default `

OpsRequest oldap-yydccx-volumeexpansion-qkglc created successfully, you can view the progress:
kbcli cluster describe-ops oldap-yydccx-volumeexpansion-qkglc -n default

➜  ~ k describe ops oldap-yydccx-volumeexpansion-qkglc
Name:         oldap-yydccx-volumeexpansion-qkglc
Namespace:    default
Labels:       app.kubernetes.io/instance=oldap-yydccx
              ops.kubeblocks.io/ops-type=VolumeExpansion
Annotations:  kubeblocks.io/reconcile: 2023-09-22T05:41:08.627369658Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2023-09-22T05:40:59Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  oldap-yydccx-volumeexpansion-
  Generation:     2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
      f:spec:
        .:
        f:clusterRef:
        f:ttlSecondsBeforeAbort:
        f:type:
        f:volumeExpansion:
          .:
          k:{"componentName":"openldap-compdef"}:
            .:
            f:componentName:
            f:volumeClaimTemplates:
              .:
              k:{"name":"data"}:
                .:
                f:name:
                f:storage:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-09-22T05:40:59Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:components:
          .:
          f:openldap-compdef:
            .:
            f:progressDetails:
        f:conditions:
          .:
          k:{"type":"Progressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"VolumeExpanding"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:lastConfiguration:
          .:
          f:components:
            .:
            f:openldap-compdef:
              .:
              f:volumeClaimTemplates:
        f:phase:
        f:progress:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-09-22T05:41:01Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:ops.kubeblocks.io/ops-type:
        f:ownerReferences:
          .:
          k:{"uid":"e4967c55-384e-42a7-ab0a-c4b982cc862e"}:
    Manager:    manager
    Operation:  Update
    Time:       2023-09-22T05:41:08Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            oldap-yydccx
    UID:             e4967c55-384e-42a7-ab0a-c4b982cc862e
  Resource Version:  635576
  UID:               1418f932-f198-4a53-8e43-7bc367597390
Spec:
  Cluster Ref:               oldap-yydccx
  Ttl Seconds Before Abort:  0
  Type:                      VolumeExpansion
  Volume Expansion:
    Component Name:  openldap-compdef
    Volume Claim Templates:
      Name:     data
      Storage:  6Gi
Status:
  Cluster Generation:  3
  Components:
    Openldap - Compdef:
      Progress Details:
        Group:       data
        Message:     Start expanding volume: PVC/data-oldap-yydccx-openldap-compdef-0 in Component: openldap-compdef
        Object Key:  PVC/data-oldap-yydccx-openldap-compdef-0
        Status:      Processing
        Group:       data
        Message:     Start expanding volume: PVC/data-oldap-yydccx-openldap-compdef-1 in Component: openldap-compdef
        Object Key:  PVC/data-oldap-yydccx-openldap-compdef-1
        Status:      Processing
  Conditions:
    Last Transition Time:  2023-09-22T05:40:59Z
    Message:               Start to process the OpsRequest: oldap-yydccx-volumeexpansion-qkglc in Cluster: oldap-yydccx
    Reason:                OpsRequestProgressingStarted
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-09-22T05:40:59Z
    Message:               OpsRequest: oldap-yydccx-volumeexpansion-qkglc is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2023-09-22T05:40:59Z
    Message:               Start to expand the volumes in Cluster: oldap-yydccx
    Reason:                VolumeExpansionStarted
    Status:                True
    Type:                  VolumeExpanding
  Last Configuration:
    Components:
      Openldap - Compdef:
        Volume Claim Templates:
          Name:     data
          Storage:  1Gi
  Phase:            Running
  Progress:         0/2
  Start Timestamp:  2023-09-22T05:40:59Z
Events:
  Type    Reason                        Age   From                    Message
  ----    ------                        ----  ----                    -------
  Normal  OpsRequestProgressingStarted  5m9s  ops-request-controller  Start to process the OpsRequest: oldap-yydccx-volumeexpansion-qkglc in Cluster: oldap-yydccx
  Normal  ValidateOpsRequestPassed      5m9s  ops-request-controller  OpsRequest: oldap-yydccx-volumeexpansion-qkglc is validated
  Normal  VolumeExpansionStarted        5m9s  ops-request-controller  Start to expand the volumes in Cluster: oldap-yydccx

➜  ~ k describe cluster oldap-yydccx
Name:         oldap-yydccx
Namespace:    default
Labels:       app.kubernetes.io/instance=oldap-yydccx
              clusterdefinition.kubeblocks.io/name=openldap
              clusterversion.kubeblocks.io/name=openldap-2.4.57
Annotations:  <none>
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-09-22T05:39:20Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  3
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:app.kubernetes.io/instance:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:backup:
          .:
          f:enabled:
          f:method:
          f:pitrEnabled:
          f:retentionPeriod:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"openldap-compdef"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-09-22T05:40:40Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"openldap-compdef"}:
            f:volumeClaimTemplates:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:      manager
    Operation:    Update
    Time:         2023-09-22T05:40:59Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:openldap-compdef:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-09-22T05:41:00Z
  Resource Version:  635489
  UID:               e4967c55-384e-42a7-ab0a-c4b982cc862e
Spec:
  Affinity:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
  Backup:
    Enabled:               false
    Method:                snapshot
    Pitr Enabled:          false
    Retention Period:      1d
  Cluster Definition Ref:  openldap
  Cluster Version Ref:     openldap-2.4.57
  Component Specs:
    Component Def Ref:  openldap-compdef
    Monitor:            true
    Name:               openldap-compdef
    No Create PDB:      false
    Replicas:           2
    Resources:
      Limits:
        Cpu:     100m
        Memory:  512Mi
      Requests:
        Cpu:               100m
        Memory:            512Mi
    Service Account Name:  kb-oldap-yydccx
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  6Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  DoNotTerminate
Status:
  Cluster Def Generation:  2
  Components:
    Openldap - Compdef:
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-09-22T05:39:53Z
  Conditions:
    Last Transition Time:  2023-09-22T05:39:20Z
    Message:               The operator has started the provisioning of Cluster: oldap-yydccx
    Observed Generation:   3
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-09-22T05:39:20Z
    Message:               Successfully applied for resources
    Observed Generation:   3
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-09-22T05:41:00Z
    Message:               pods are not ready in Components: [openldap-compdef], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-09-22T05:41:00Z
    Message:               pods are unavailable in Components: [openldap-compdef], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     3
  Phase:                   Updating
Events:
  Type     Reason                    Age                    From                Message
  ----     ------                    ----                   ----                -------
  Normal   ComponentPhaseTransition  7m33s                  cluster-controller  Create a new component
  Normal   Running                   7m                     cluster-controller  Cluster: oldap-yydccx is ready, current phase is Running
  Normal   ComponentPhaseTransition  7m                     cluster-controller  component is Running
  Normal   AllReplicasReady          7m                     cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              7m                     cluster-controller  Cluster: oldap-yydccx is ready, current phase is Running
  Normal   ApplyResourcesSucceed     5m53s (x3 over 7m33s)  cluster-controller  Successfully applied for resources
  Normal   PreCheckSucceed           5m53s (x3 over 7m33s)  cluster-controller  The operator has started the provisioning of Cluster: oldap-yydccx
  Normal   ComponentPhaseTransition  5m53s                  cluster-controller  component is Updating
  Warning  ReplicasNotReady          5m53s                  cluster-controller  pods are not ready in Components: [openldap-compdef], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        5m53s                  cluster-controller  pods are unavailable in Components: [openldap-compdef], refer to related component message in Cluster.status.components

➜  ~ k logs oldap-yydccx-openldap-compdef-0
***  INFO   | 2023-09-22 05:39:38 | CONTAINER_LOG_LEVEL = 3 (info)
***  INFO   | 2023-09-22 05:39:38 | Search service in CONTAINER_SERVICE_DIR = /container/service :
***  INFO   | 2023-09-22 05:39:38 | link /container/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools
***  INFO   | 2023-09-22 05:39:38 | link /container/service/slapd/startup.sh to /container/run/startup/slapd
***  INFO   | 2023-09-22 05:39:38 | link /container/service/slapd/process.sh to /container/run/process/slapd/run
***  INFO   | 2023-09-22 05:39:38 | Environment files will be proccessed in this order :
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.yaml
/container/environment/99-default/default.startup.yaml

To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
***  INFO   | 2023-09-22 05:39:39 | Running /container/run/startup/:ssl-tools...
***  INFO   | 2023-09-22 05:39:39 | Running /container/run/startup/slapd...
***  INFO   | 2023-09-22 05:39:39 | openldap user and group adjustments
***  INFO   | 2023-09-22 05:39:39 | get current openldap uid/gid info inside container
***  INFO   | 2023-09-22 05:39:39 | -------------------------------------
***  INFO   | 2023-09-22 05:39:39 | openldap GID/UID
***  INFO   | 2023-09-22 05:39:39 | -------------------------------------
***  INFO   | 2023-09-22 05:39:39 | User uid: 911
***  INFO   | 2023-09-22 05:39:39 | User gid: 911
***  INFO   | 2023-09-22 05:39:39 | uid/gid changed: false
***  INFO   | 2023-09-22 05:39:40 | -------------------------------------
***  INFO   | 2023-09-22 05:39:40 | updating file uid/gid ownership
***  INFO   | 2023-09-22 05:39:40 | Database and config directory are empty...
***  INFO   | 2023-09-22 05:39:40 | Init new ldap server...
  Backing up /etc/ldap/slapd.d in /var/backups/slapd-2.4.57+dfsg-1~bpo10+1... done.
  Creating initial configuration... done.
  Creating LDAP directory... done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
***  INFO   | 2023-09-22 05:39:45 | Start OpenLDAP...
***  INFO   | 2023-09-22 05:39:45 | Waiting for OpenLDAP to start...
***  INFO   | 2023-09-22 05:39:45 | Add bootstrap schemas...
config file testing succeeded
***  INFO   | 2023-09-22 05:39:49 | Add image bootstrap ldif...
***  INFO   | 2023-09-22 05:39:51 | Add custom bootstrap ldif...
***  INFO   | 2023-09-22 05:39:52 | Disable replication config...
***  INFO   | 2023-09-22 05:39:52 | Stop OpenLDAP...
***  INFO   | 2023-09-22 05:39:52 | First start is done...
***  INFO   | 2023-09-22 05:39:52 | Remove file /container/environment/99-default/default.startup.yaml
***  INFO   | 2023-09-22 05:39:52 | Environment files will be proccessed in this order :
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.yaml

To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
***  INFO   | 2023-09-22 05:39:52 | Running /container/run/process/slapd/run...
650d28a9 @(#) $OpenLDAP: slapd 2.4.57+dfsg-1~bpo10+1 (Jan 30 2021 06:59:51) $
    Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
650d28a9 slapd starting
650d28c3 conn=1000 fd=12 ACCEPT from IP=10.92.1.15:36698 (IP=0.0.0.0:389)
650d28c3 conn=1000 op=0 BIND dn="cn=admin,dc=kubeblocks,dc=io" method=128
650d28c3 conn=1000 op=0 BIND dn="cn=admin,dc=kubeblocks,dc=io" mech=SIMPLE ssf=0
650d28c3 conn=1000 op=0 RESULT tag=97 err=0 text=
650d28c3 conn=1000 op=1 SRCH base="dc=kubeblocks,dc=io" scope=2 deref=0 filter="(objectClass=*)"
650d28c3 conn=1000 op=1 SEARCH RESULT tag=101 err=0 nentries=1 text=
650d28c3 conn=1000 op=2 UNBIND
650d28c3 conn=1000 fd=12 closed
650d28e9 conn=1001 fd=12 ACCEPT from IP=10.92.1.15:49300 (IP=0.0.0.0:389)
650d28e9 conn=1001 op=0 BIND dn="cn=admin,dc=kubeblocks,dc=io" method=128
650d28e9 conn=1001 op=0 BIND dn="cn=admin,dc=kubeblocks,dc=io" mech=SIMPLE ssf=0
650d28e9 conn=1001 op=0 RESULT tag=97 err=0 text=
650d28e9 conn=1001 op=1 SRCH base="dc=kubeblocks,dc=io" scope=2 deref=0 filter="(objectClass=*)"
650d28e9 conn=1001 op=1 SEARCH RESULT tag=101 err=0 nentries=1 text=
650d28e9 conn=1001 op=2 UNBIND
650d28e9 conn=1001 fd=12 closed

➜  ~ k logs oldap-yydccx-openldap-compdef-1
***  INFO   | 2023-09-22 05:39:53 | CONTAINER_LOG_LEVEL = 3 (info)
***  INFO   | 2023-09-22 05:39:53 | Search service in CONTAINER_SERVICE_DIR = /container/service :
***  INFO   | 2023-09-22 05:39:53 | link /container/service/:ssl-tools/startup.sh to /container/run/startup/:ssl-tools
***  INFO   | 2023-09-22 05:39:53 | link /container/service/slapd/startup.sh to /container/run/startup/slapd
***  INFO   | 2023-09-22 05:39:53 | link /container/service/slapd/process.sh to /container/run/process/slapd/run
***  INFO   | 2023-09-22 05:39:53 | Environment files will be proccessed in this order :
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.yaml
/container/environment/99-default/default.startup.yaml

To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
***  INFO   | 2023-09-22 05:39:53 | Running /container/run/startup/:ssl-tools...
***  INFO   | 2023-09-22 05:39:53 | Running /container/run/startup/slapd...
***  INFO   | 2023-09-22 05:39:53 | openldap user and group adjustments
***  INFO   | 2023-09-22 05:39:53 | get current openldap uid/gid info inside container
***  INFO   | 2023-09-22 05:39:53 | -------------------------------------
***  INFO   | 2023-09-22 05:39:54 | openldap GID/UID
***  INFO   | 2023-09-22 05:39:54 | -------------------------------------
***  INFO   | 2023-09-22 05:39:54 | User uid: 911
***  INFO   | 2023-09-22 05:39:54 | User gid: 911
***  INFO   | 2023-09-22 05:39:54 | uid/gid changed: false
***  INFO   | 2023-09-22 05:39:54 | -------------------------------------
***  INFO   | 2023-09-22 05:39:54 | updating file uid/gid ownership
***  INFO   | 2023-09-22 05:39:54 | Database and config directory are empty...
***  INFO   | 2023-09-22 05:39:54 | Init new ldap server...
  Backing up /etc/ldap/slapd.d in /var/backups/slapd-2.4.57+dfsg-1~bpo10+1... done.
  Creating initial configuration... done.
  Creating LDAP directory... done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
***  INFO   | 2023-09-22 05:39:58 | Start OpenLDAP...
***  INFO   | 2023-09-22 05:39:59 | Waiting for OpenLDAP to start...
***  INFO   | 2023-09-22 05:39:59 | Add bootstrap schemas...
config file testing succeeded
***  INFO   | 2023-09-22 05:40:03 | Add image bootstrap ldif...
***  INFO   | 2023-09-22 05:40:05 | Add custom bootstrap ldif...
***  INFO   | 2023-09-22 05:40:05 | Disable replication config...
***  INFO   | 2023-09-22 05:40:05 | Stop OpenLDAP...
***  INFO   | 2023-09-22 05:40:05 | First start is done...
***  INFO   | 2023-09-22 05:40:06 | Remove file /container/environment/99-default/default.startup.yaml
***  INFO   | 2023-09-22 05:40:06 | Environment files will be proccessed in this order :
Caution: previously defined variables will not be overriden.
/container/environment/99-default/default.yaml

To see how this files are processed and environment variables values,
run this container with '--loglevel debug'
***  INFO   | 2023-09-22 05:40:06 | Running /container/run/process/slapd/run...
650d28b6 @(#) $OpenLDAP: slapd 2.4.57+dfsg-1~bpo10+1 (Jan 30 2021 06:59:51) $
    Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
650d28b6 slapd starting
ahjing99 commented 1 year ago 

➜  ~ k logs kubeblocks-66fb6c98d8-bk5wn -n kb-system
Defaulted container "manager" out of: manager, tools (init), datascript (init)
2023-09-21T10:30:57.974Z    INFO    setup   config file: /etc/kubeblocks/config.yaml
2023-09-21T10:30:57.975Z    INFO    setup   config settings: map[apps:true backup_pv_configmap_name: backup_pv_configmap_namespace: backup_pvc_create_policy: backup_pvc_init_capacity: backup_pvc_name: backup_pvc_storage_class: cert_dir:/tmp/k8s-webhook-server/serving-certs cm_namespace:kb-system cm_recon_retry_duration_ms:1000 config_manager_grpc_port:9901 config_manager_log_level:info data_plane_affinity:{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"kb-data","operator":"In","values":["true"]}]},"weight":100}]}} data_plane_tolerations:[{"effect":"NoSchedule","key":"kb-data","operator":"Equal","value":"true"}] default_storage_class:kb-default-sc enable_debug_sysaccounts:false enablerbacmanager:true extensions:true health_probe_bind_address::8081 kubeblocks_addon_helm_install_options:--atomic --cleanup-on-fail --wait --insecure-skip-tls-verify kubeblocks_addon_helm_uninstall_options:[] kubeblocks_addon_sa_name:kubeblocks-addon-installer kubeblocks_datascript_clients_image:registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-datascript:0.7.0-alpha.15 kubeblocks_serviceaccount_name:kubeblocks kubeblocks_tools_image:registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-alpha.15 kubeconfig: kubernetes_cluster_domain:cluster.local leader_elect:true leader_elect_id:001c317f maxconcurrentreconciles_addon:4 maxconcurrentreconciles_clusterdef:2 maxconcurrentreconciles_clusterversion:2 metrics_bind_address::8080 pod_min_ready_seconds:10 probe_service_grpc_port:50001 probe_service_http_port:3501 probe_service_log_level:info replicated_state_machine:true storage:true volumesnapshot:true volumesnapshot_api_beta:true workloads:true zap_devel:false zap_encoder:console zap_log_level: zap_stacktrace_level: zap_time_encoding:iso8601]
2023-09-21T10:30:57.995Z    INFO    controller-runtime.metrics  Metrics server is starting to listen    {"addr": ":8080"}
2023-09-21T10:30:57.999Z    INFO    setup   starting manager
2023-09-21T10:30:58.000Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-09-21T10:30:58.000Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0921 10:30:58.000553       1 leaderelection.go:248] attempting to acquire leader lease kb-system/001c317f.kubeblocks.io...
I0921 10:30:58.009125       1 leaderelection.go:258] successfully acquired lease kb-system/001c317f.kubeblocks.io
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.StatefulSet"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Deployment"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.ReplicatedStateMachine"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Service"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ConfigMap"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.PodDisruptionBudget"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.BackupPolicy"}
2023-09-21T10:30:58.009Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Backup"}
2023-09-21T10:30:58.010Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-09-21T10:30:58.010Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ClusterRoleBinding"}
2023-09-21T10:30:58.010Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.RoleBinding"}
2023-09-21T10:30:58.010Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.ServiceAccount"}
2023-09-21T10:30:58.010Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Pod"}
2023-09-21T10:30:58.010Z    INFO    Starting Controller {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-09-21T10:30:58.010Z    INFO    Starting EventSource    {"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "source": "kind source: *v1alpha1.ClusterVersion"}
2023-09-21T10:30:58.010Z    INFO    Starting Controller {"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion"}
2023-09-21T10:30:58.011Z    INFO    Starting EventSource    {"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "source": "kind source: *v1alpha1.ClusterDefinition"}
2023-09-21T10:30:58.011Z    INFO    Starting Controller {"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition"}
2023-09-21T10:30:58.011Z    INFO    Starting EventSource    {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1alpha1.ConfigConstraint"}
2023-09-21T10:30:58.011Z    INFO    Starting EventSource    {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "source": "kind source: *v1.ConfigMap"}
2023-09-21T10:30:58.011Z    INFO    Starting Controller {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint"}
2023-09-21T10:30:58.012Z    INFO    Starting EventSource    {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.OpsRequest"}
2023-09-21T10:30:58.012Z    INFO    Starting EventSource    {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.Cluster"}
2023-09-21T10:30:58.012Z    INFO    Starting EventSource    {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "source": "kind source: *v1alpha1.Backup"}
2023-09-21T10:30:58.012Z    INFO    Starting Controller {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest"}
2023-09-21T10:30:58.012Z    INFO    Starting EventSource    {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "source": "kind source: *v1alpha1.Configuration"}
2023-09-21T10:30:58.012Z    INFO    Starting EventSource    {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "source": "kind source: *v1.ConfigMap"}
2023-09-21T10:30:58.012Z    INFO    Starting Controller {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration"}
2023-09-21T10:30:58.013Z    INFO    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2023-09-21T10:30:58.014Z    INFO    Starting Controller {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2023-09-21T10:30:58.014Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1alpha1.Cluster"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Secret"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "source": "kind source: *v1.Job"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster"}
2023-09-21T10:30:58.014Z    INFO    Starting EventSource    {"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor", "source": "kind source: *v1alpha1.ServiceDescriptor"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor"}
2023-09-21T10:30:58.013Z    INFO    Starting EventSource    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "source": "kind source: *v1.Event"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "event", "controllerGroup": "", "controllerKind": "Event"}
2023-09-21T10:30:58.014Z    INFO    Starting EventSource    {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "source": "kind source: *v1alpha1.ComponentClassDefinition"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition"}
2023-09-21T10:30:58.014Z    INFO    Starting EventSource    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim"}
2023-09-21T10:30:58.014Z    INFO    Starting EventSource    {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1alpha1.Addon"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "source": "kind source: *v1.Job"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon"}
2023-09-21T10:30:58.014Z    INFO    Starting EventSource    {"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "source": "kind source: *v1alpha1.StorageProvider"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "source": "kind source: *v1.CSIDriver"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider"}
2023-09-21T10:30:58.015Z    INFO    Starting EventSource    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1alpha1.ReplicatedStateMachine"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.StatefulSet"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.Job"}
2023-09-21T10:30:58.016Z    INFO    Starting EventSource    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "source": "kind source: *v1.Pod"}
2023-09-21T10:30:58.016Z    INFO    Starting Controller {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine"}
2023-09-21T10:30:58.218Z    INFO    Starting workers    {"controller": "clusterdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterDefinition", "worker count": 2}
2023-09-21T10:30:58.222Z    INFO    Starting workers    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "worker count": 1}
2023-09-21T10:30:58.222Z    INFO    Starting workers    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "worker count": 1}
2023-09-21T10:30:58.231Z    INFO    Starting workers    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2023-09-21T10:30:58.231Z    INFO    Starting workers    {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "worker count": 1}
2023-09-21T10:30:58.231Z    INFO    Starting workers    {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "worker count": 1}
2023-09-21T10:30:58.232Z    INFO    Starting workers    {"controller": "configconstraint", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ConfigConstraint", "worker count": 1}
2023-09-21T10:30:58.232Z    INFO    Starting workers    {"controller": "storageprovider", "controllerGroup": "storage.kubeblocks.io", "controllerKind": "StorageProvider", "worker count": 1}
2023-09-21T10:30:58.232Z    INFO    Starting workers    {"controller": "addon", "controllerGroup": "extensions.kubeblocks.io", "controllerKind": "Addon", "worker count": 4}
2023-09-21T10:30:58.234Z    INFO    Starting workers    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-09-21T10:30:58.234Z    INFO    Starting workers    {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "worker count": 1}
2023-09-21T10:30:58.234Z    INFO    Starting workers    {"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "worker count": 2}
2023-09-21T10:30:58.234Z    INFO    Starting workers    {"controller": "servicedescriptor", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ServiceDescriptor", "worker count": 1}
2023-09-21T10:30:58.235Z    INFO    Starting workers    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "worker count": 1}
2023-09-21T10:30:58.235Z    INFO    Starting workers    {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "worker count": 1}
2023-09-21T10:31:20.682Z    INFO    class general-1c1g does not conform to any constraints  {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.mysql.mysql"}, "namespace": "", "name": "kb.classes.default.mysql.mysql", "reconcileID": "c2c78723-052b-4446-b4cf-59205c882e40", "classDefinition": "/kb.classes.default.mysql.mysql"}
2023-09-21T10:31:20.684Z    INFO    class general-2c2g does not conform to any constraints  {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.mysql.mysql"}, "namespace": "", "name": "kb.classes.default.mysql.mysql", "reconcileID": "8d0fcba4-37e5-4c11-88ad-19c3a4e21a28", "classDefinition": "/kb.classes.default.mysql.mysql"}
2023-09-21T10:31:22.951Z    INFO    KubeAPIWarningLogger    child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning
2023-09-21T10:31:25.168Z    INFO    class general-2c4g does not conform to any constraints  {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.apecloud-mysql.mysql"}, "namespace": "", "name": "kb.classes.default.apecloud-mysql.mysql", "reconcileID": "5927e789-922d-46bc-a475-063d842ee1bf", "classDefinition": "/kb.classes.default.apecloud-mysql.mysql"}
2023-09-21T10:31:25.171Z    INFO    class general-32c128g does not conform to any constraints   {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.apecloud-mysql.mysql"}, "namespace": "", "name": "kb.classes.default.apecloud-mysql.mysql", "reconcileID": "62666d65-91be-4460-a667-5026d6e10783", "classDefinition": "/kb.classes.default.apecloud-mysql.mysql"}
2023-09-21T10:33:01.870Z    ERROR   ConfigurationReconcile.mysql-replication-config     {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "fe6f6488-6c08-4a36-81b9-80d8b59b3dea", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {mysql-replication-config  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.870Z    ERROR   ConfigurationReconcile.agamotto-configuration       {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "fe6f6488-6c08-4a36-81b9-80d8b59b3dea", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {agamotto-configuration  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.906Z    ERROR   ConfigurationReconcile.mysql-replication-config     {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "ef48a665-0e8a-4f1e-a9a6-f0148cfc589f", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {mysql-replication-config  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.906Z    ERROR   ConfigurationReconcile.agamotto-configuration       {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "ef48a665-0e8a-4f1e-a9a6-f0148cfc589f", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {agamotto-configuration  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.924Z    ERROR   ConfigurationReconcile.mysql-replication-config     {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "c099b880-255b-45f1-afaf-40680b646a4f", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {mysql-replication-config  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.924Z    ERROR   ConfigurationReconcile.agamotto-configuration       {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "c099b880-255b-45f1-afaf-40680b646a4f", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {agamotto-configuration  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.925Z    ERROR   ConfigurationReconcile.mysql-replication-config     {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "58b5e4ea-2fe2-48be-b206-4a3f4c4b682b", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {mysql-replication-config  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:01.926Z    ERROR   ConfigurationReconcile.agamotto-configuration       {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "58b5e4ea-2fe2-48be-b206-4a3f4c4b682b", "configuration": "default/mysql1-mysql", "error": "configSpec phase is not ready and pass: {agamotto-configuration  <nil> map[]}"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.fromItemStatus
    /src/controllers/apps/configuration/configuration_controller.go:191
github.com/apecloud/kubeblocks/controllers/apps/configuration.(*ConfigurationReconciler).Reconcile
    /src/controllers/apps/configuration/configuration_controller.go:79
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:02.034Z    INFO    update cluster status after applying resources, generation: 1   {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mysql1","namespace":"default"}, "namespace": "default", "name": "mysql1", "reconcileID": "59ed9cac-5a01-4c12-b354-af23ce958698", "cluster": "default/mysql1"}
2023-09-21T10:33:02.131Z    INFO    DAG: |->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "e890ff5e-cf1a-4f3e-bdaf-989bd6c2e5d8", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:02.133Z    INFO    KubeAPIWarningLogger    unknown field "spec.service.metadata.creationTimestamp"
2023-09-21T10:33:02.133Z    INFO    KubeAPIWarningLogger    unknown field "spec.template.metadata.creationTimestamp"
2023-09-21T10:33:02.133Z    INFO    KubeAPIWarningLogger    unknown field "spec.volumeClaimTemplates[0].metadata.creationTimestamp"
2023-09-21T10:33:02.180Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: CREATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: CREATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: CREATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: CREATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "38b1b7dd-ccb4-4f4a-a085-f3b9cd6bdf4a", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:02.505Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "0fc4b59c-588b-4ce1-a975-22e1473fdccd", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:02.754Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "b6f345b4-a6e4-4414-9d19-190991260432", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:02.953Z    ERROR   update *v1.StatefulSet error: mysql1-mysql  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "b6f345b4-a6e4-4414-9d19-190991260432", "ReplicatedStateMachine": "default/mysql1-mysql", "error": "Operation cannot be fulfilled on statefulsets.apps \"mysql1-mysql\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/apecloud/kubeblocks/internal/controller/rsm.(*PlanBuilder).rsmWalkFunc
    /src/internal/controller/rsm/plan_builder.go:126
github.com/apecloud/kubeblocks/internal/controller/graph.(*DAG).WalkReverseTopoOrder
    /src/internal/controller/graph/dag.go:168
github.com/apecloud/kubeblocks/internal/controller/rsm.(*Plan).Execute
    /src/internal/controller/rsm/plan_builder.go:101
github.com/apecloud/kubeblocks/controllers/workloads.(*ReplicatedStateMachineReconciler).Reconcile
    /src/controllers/workloads/replicatedstatemachine_controller.go:129
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:02.953Z    INFO    Operation cannot be fulfilled on statefulsets.apps "mysql1-mysql": the object has been modified; please apply your changes to the latest version and try again  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "b6f345b4-a6e4-4414-9d19-190991260432", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:02.953Z    ERROR   Reconciler error    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "b6f345b4-a6e4-4414-9d19-190991260432", "error": "Operation cannot be fulfilled on statefulsets.apps \"mysql1-mysql\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-21T10:33:02.960Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "29f2c154-0191-4ee9-a3a6-68936d88b4f6", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:03.085Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "2bfcf4f1-91f1-4557-b6d9-4effea09a236", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:03.281Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "e4451331-90f3-4c5c-84b2-11bbbb2e61a4", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:09.771Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "72cf882e-ca8b-4703-bc06-9c649888a372", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:10.035Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "b538f312-388d-45be-8e12-765614f2c1c6", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:10.210Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "28f8967b-4b82-4425-ba52-46acf2a0156a", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:22.440Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "7ea5b9b0-6ac7-4dfc-a12e-f2ca490545a4", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:33:23.646Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "20ff5d25-36cb-4b4b-ac1f-9e927e546712", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:34:06.744Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e37f8e00267e","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e37f8e00267e", "reconcileID": "85130e5c-b0d4-4624-ba8e-a8ced286ba71", "event": "default/mysql1-mysql-0.1786e37f8e00267e", "message": "Pulling image \"apecloud/kubeblocks-tools:latest\""}
2023-09-21T10:34:06.744Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e37f8e00267e","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e37f8e00267e", "reconcileID": "85130e5c-b0d4-4624-ba8e-a8ced286ba71", "event": "default/mysql1-mysql-0.1786e37f8e00267e", "message": "Pulling image \"apecloud/kubeblocks-tools:latest\""}
2023-09-21T10:34:07.221Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e37faa7325c4","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e37faa7325c4", "reconcileID": "209d4672-bf46-4677-ae6e-05b306c52f9e", "event": "default/mysql1-mysql-1.1786e37faa7325c4", "message": "Pulling image \"apecloud/kubeblocks-tools:latest\""}
2023-09-21T10:34:07.221Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e37faa7325c4","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e37faa7325c4", "reconcileID": "209d4672-bf46-4677-ae6e-05b306c52f9e", "event": "default/mysql1-mysql-1.1786e37faa7325c4", "message": "Pulling image \"apecloud/kubeblocks-tools:latest\""}
2023-09-21T10:34:13.996Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e3813e1314ea","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e3813e1314ea", "reconcileID": "d891c130-34d0-4905-a242-34f4d74b63df", "event": "default/mysql1-mysql-0.1786e3813e1314ea", "message": "Successfully pulled image \"apecloud/kubeblocks-tools:latest\" in 7.248949606s (7.248968187s including waiting)"}
2023-09-21T10:34:13.996Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e3813e1314ea","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e3813e1314ea", "reconcileID": "d891c130-34d0-4905-a242-34f4d74b63df", "event": "default/mysql1-mysql-0.1786e3813e1314ea", "message": "Successfully pulled image \"apecloud/kubeblocks-tools:latest\" in 7.248949606s (7.248968187s including waiting)"}
2023-09-21T10:34:14.028Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e38140539852","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e38140539852", "reconcileID": "93a4884c-22bb-4dd6-9be9-3ba2255e93c5", "event": "default/mysql1-mysql-0.1786e38140539852", "message": "Created container kb-role-probe"}
2023-09-21T10:34:14.028Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e38140539852","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e38140539852", "reconcileID": "93a4884c-22bb-4dd6-9be9-3ba2255e93c5", "event": "default/mysql1-mysql-0.1786e38140539852", "message": "Created container kb-role-probe"}
2023-09-21T10:34:14.147Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e3814707775f","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e3814707775f", "reconcileID": "a730f933-e465-4e33-b1f9-ac4b61c79420", "event": "default/mysql1-mysql-0.1786e3814707775f", "message": "Started container kb-role-probe"}
2023-09-21T10:34:14.147Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-0.1786e3814707775f","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-0.1786e3814707775f", "reconcileID": "a730f933-e465-4e33-b1f9-ac4b61c79420", "event": "default/mysql1-mysql-0.1786e3814707775f", "message": "Started container kb-role-probe"}
2023-09-21T10:34:14.760Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "f2376056-d00e-4f46-9186-98f46b0c0c2f", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:34:14.979Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "0c17a44a-83c7-47a4-8ce5-25e6ae938e8b", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:34:15.098Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "ed85798b-4c95-443b-8a13-a0ab64f09bbf", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:34:17.193Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e381fd05d9b7","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e381fd05d9b7", "reconcileID": "d691d616-d54d-4e32-a17d-8f4407011621", "event": "default/mysql1-mysql-1.1786e381fd05d9b7", "message": "Successfully pulled image \"apecloud/kubeblocks-tools:latest\" in 9.975175547s (9.975240869s including waiting)"}
2023-09-21T10:34:17.194Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e381fd05d9b7","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e381fd05d9b7", "reconcileID": "d691d616-d54d-4e32-a17d-8f4407011621", "event": "default/mysql1-mysql-1.1786e381fd05d9b7", "message": "Successfully pulled image \"apecloud/kubeblocks-tools:latest\" in 9.975175547s (9.975240869s including waiting)"}
2023-09-21T10:34:17.224Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e381fee33405","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e381fee33405", "reconcileID": "951359dd-0f9a-4cc0-9505-d2b3947ab839", "event": "default/mysql1-mysql-1.1786e381fee33405", "message": "Created container kb-role-probe"}
2023-09-21T10:34:17.225Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e381fee33405","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e381fee33405", "reconcileID": "951359dd-0f9a-4cc0-9505-d2b3947ab839", "event": "default/mysql1-mysql-1.1786e381fee33405", "message": "Created container kb-role-probe"}
2023-09-21T10:34:17.328Z    INFO    parser Readiness probe event message failed {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e38205069ccf","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e38205069ccf", "reconcileID": "be9ae319-992b-4702-b928-83d250c7f7b1", "event": "default/mysql1-mysql-1.1786e38205069ccf", "message": "Started container kb-role-probe"}
2023-09-21T10:34:17.328Z    INFO    parse probe event message failed    {"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"mysql1-mysql-1.1786e38205069ccf","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql-1.1786e38205069ccf", "reconcileID": "be9ae319-992b-4702-b928-83d250c7f7b1", "event": "default/mysql1-mysql-1.1786e38205069ccf", "message": "Started container kb-role-probe"}
2023-09-21T10:34:17.902Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "36c212eb-5a09-4ac2-a3f5-f7a7587126f5", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:34:18.183Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "317d3ff7-91ee-453e-8982-fb77bd457896", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-21T10:34:18.322Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "f738fbc8-90d3-408f-8f83-07ef6f17bee9", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-22T05:10:54.738Z    INFO    class mo-48c384g does not conform to any constraints    {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.mysql.mysql"}, "namespace": "", "name": "kb.classes.default.mysql.mysql", "reconcileID": "c8a66664-4627-4660-8b12-e6b495a4abaf", "classDefinition": "/kb.classes.default.mysql.mysql"}
2023-09-22T05:10:54.740Z    INFO    class mo-4c32g does not conform to any constraints  {"controller": "componentclassdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentClassDefinition", "ComponentClassDefinition": {"name":"kb.classes.default.apecloud-mysql.mysql"}, "namespace": "", "name": "kb.classes.default.apecloud-mysql.mysql", "reconcileID": "685b9db3-9a9d-48ff-817f-f3aa87782873", "classDefinition": "/kb.classes.default.apecloud-mysql.mysql"}
2023-09-22T05:39:20.667Z    INFO    update cluster status after applying resources, generation: 1   {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"oldap-yydccx","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx", "reconcileID": "fcb238e9-cb02-4759-8f49-cb5b62806e55", "cluster": "default/oldap-yydccx"}
2023-09-22T05:39:20.729Z    INFO    DAG: |->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "72e79eb5-9b35-47fd-a559-b4574175d3d2", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:20.777Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: CREATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: CREATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: CREATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: CREATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "60273153-71b9-4de0-86b8-a9a3027c5349", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:21.732Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "670b4173-707e-488e-af25-7bfc04669998", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:21.938Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "a584c365-8db6-4571-87fb-b31f98ccb5b7", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:23.449Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "f3ed3a7f-868d-49bd-a150-5be2f0e5d64b", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:23.604Z    ERROR   update *v1.StatefulSet error: oldap-yydccx-openldap-compdef {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "f3ed3a7f-868d-49bd-a150-5be2f0e5d64b", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef", "error": "Operation cannot be fulfilled on statefulsets.apps \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/apecloud/kubeblocks/internal/controller/rsm.(*PlanBuilder).rsmWalkFunc
    /src/internal/controller/rsm/plan_builder.go:126
github.com/apecloud/kubeblocks/internal/controller/graph.(*DAG).WalkReverseTopoOrder
    /src/internal/controller/graph/dag.go:168
github.com/apecloud/kubeblocks/internal/controller/rsm.(*Plan).Execute
    /src/internal/controller/rsm/plan_builder.go:101
github.com/apecloud/kubeblocks/controllers/workloads.(*ReplicatedStateMachineReconciler).Reconcile
    /src/controllers/workloads/replicatedstatemachine_controller.go:129
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:23.604Z    INFO    Operation cannot be fulfilled on statefulsets.apps "oldap-yydccx-openldap-compdef": the object has been modified; please apply your changes to the latest version and try again {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "f3ed3a7f-868d-49bd-a150-5be2f0e5d64b", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:23.605Z    ERROR   Reconciler error    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "f3ed3a7f-868d-49bd-a150-5be2f0e5d64b", "error": "Operation cannot be fulfilled on statefulsets.apps \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:23.624Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "0e5d6204-be48-4feb-846b-e8e8636baf11", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:23.814Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "1964c7d2-2bf2-415b-9c96-d7f24ccf8360", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:23.925Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "7720286a-1ee9-429d-966e-8a1942ef3f72", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:30.944Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "a1915674-832a-42d9-b6c4-4f3bf0171bf6", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:31.085Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "a84c0c66-d215-4897-ab9c-6b430909163c", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:31.165Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "5f3c35db-661f-4990-8907-b2307ffcb347", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:39.103Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "30c9f437-226a-4cb8-8d52-1fe3fd3f0802", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:39.276Z    ERROR   update *v1.StatefulSet error: oldap-yydccx-openldap-compdef {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "30c9f437-226a-4cb8-8d52-1fe3fd3f0802", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef", "error": "Operation cannot be fulfilled on statefulsets.apps \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/apecloud/kubeblocks/internal/controller/rsm.(*PlanBuilder).rsmWalkFunc
    /src/internal/controller/rsm/plan_builder.go:126
github.com/apecloud/kubeblocks/internal/controller/graph.(*DAG).WalkReverseTopoOrder
    /src/internal/controller/graph/dag.go:168
github.com/apecloud/kubeblocks/internal/controller/rsm.(*Plan).Execute
    /src/internal/controller/rsm/plan_builder.go:101
github.com/apecloud/kubeblocks/controllers/workloads.(*ReplicatedStateMachineReconciler).Reconcile
    /src/controllers/workloads/replicatedstatemachine_controller.go:129
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:39.276Z    INFO    Operation cannot be fulfilled on statefulsets.apps "oldap-yydccx-openldap-compdef": the object has been modified; please apply your changes to the latest version and try again {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "30c9f437-226a-4cb8-8d52-1fe3fd3f0802", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:39.276Z    ERROR   Reconciler error    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "30c9f437-226a-4cb8-8d52-1fe3fd3f0802", "error": "Operation cannot be fulfilled on statefulsets.apps \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:39.285Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "4e565253-3c02-416f-a5e1-3f66eccfd50b", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:39.388Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "2cd39dfe-22df-4f26-b78d-c3497cabdbb6", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:39.512Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "bef9e4dd-8995-422e-b2ef-730aaf8f97b1", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:43.307Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "d2034a32-0f8f-434e-a0cb-da797c88152f", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:43.950Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "1f7557f2-b50d-47b1-82cc-a32e7ef17e83", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:44.120Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "dcfd5d15-39f7-4c4c-a915-04e1060fd9b2", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:52.868Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "b2d74eb2-4c35-45a6-ab30-73016c109269", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:52.920Z    ERROR   update *v1.StatefulSet error: oldap-yydccx-openldap-compdef {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "b2d74eb2-4c35-45a6-ab30-73016c109269", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef", "error": "Operation cannot be fulfilled on statefulsets.apps \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/apecloud/kubeblocks/internal/controller/rsm.(*PlanBuilder).rsmWalkFunc
    /src/internal/controller/rsm/plan_builder.go:126
github.com/apecloud/kubeblocks/internal/controller/graph.(*DAG).WalkReverseTopoOrder
    /src/internal/controller/graph/dag.go:168
github.com/apecloud/kubeblocks/internal/controller/rsm.(*Plan).Execute
    /src/internal/controller/rsm/plan_builder.go:101
github.com/apecloud/kubeblocks/controllers/workloads.(*ReplicatedStateMachineReconciler).Reconcile
    /src/controllers/workloads/replicatedstatemachine_controller.go:129
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:52.920Z    INFO    Operation cannot be fulfilled on statefulsets.apps "oldap-yydccx-openldap-compdef": the object has been modified; please apply your changes to the latest version and try again {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "b2d74eb2-4c35-45a6-ab30-73016c109269", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:52.920Z    ERROR   Reconciler error    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "b2d74eb2-4c35-45a6-ab30-73016c109269", "error": "Operation cannot be fulfilled on statefulsets.apps \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:52.927Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "6157b57a-5de2-4480-a8df-7cd72ffa07e9", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:53.054Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "c1bd0676-8618-4664-adcc-69d842d1c4cb", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:53.143Z    INFO    Operation cannot be fulfilled on replicatedstatemachines.workloads.kubeblocks.io "oldap-yydccx-openldap-compdef": the object has been modified; please apply your changes to the latest version and try again   {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "c1bd0676-8618-4664-adcc-69d842d1c4cb", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:53.146Z    ERROR   Reconciler error    {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "c1bd0676-8618-4664-adcc-69d842d1c4cb", "error": "Operation cannot be fulfilled on replicatedstatemachines.workloads.kubeblocks.io \"oldap-yydccx-openldap-compdef\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235
2023-09-22T05:39:53.159Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "4a8d4f8c-c916-4fcc-be81-41c3ab3274ce", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:39:53.361Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "81d1b7e7-0a50-436d-85b9-897329858493", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:40:41.066Z    INFO    update cluster status after applying resources, generation: 2   {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"oldap-yydccx","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx", "reconcileID": "ed01fc03-e6c5-475e-8ec6-f345eb575eec", "cluster": "default/oldap-yydccx"}
2023-09-22T05:40:41.141Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "e49836fe-c58b-4d69-bea1-2b05d04f4d64", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:40:41.277Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "c28e946f-1c5a-4e17-b254-39a99c06b132", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:40:59.817Z    INFO    update cluster status after applying resources, generation: 3   {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"oldap-yydccx","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx", "reconcileID": "d02a8e71-36e2-41aa-afd6-082819d6ffa9", "cluster": "default/oldap-yydccx"}
2023-09-22T05:40:59.935Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "ec4ee42b-fce3-4d21-98a7-9701b506c697", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:41:00.382Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "f72399b0-dbb8-416e-a6d3-675f41085bd9", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:48:34.103Z    INFO    DAG: |->{obj:*v1.Service, name: mysql1-mysql-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: mysql1-mysql-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: mysql1-mysql, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: mysql1-mysql, immutable: false, orphan: false, action: STATUS}  {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"mysql1-mysql","namespace":"default"}, "namespace": "default", "name": "mysql1-mysql", "reconcileID": "730e295a-cc84-4398-b975-e94c1d1c7dd0", "ReplicatedStateMachine": "default/mysql1-mysql"}
2023-09-22T05:48:34.199Z    INFO    DAG: |->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "4907c187-e381-4854-8904-7cebe1d7d317", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
2023-09-22T05:48:34.267Z    INFO    DAG: |->{obj:*v1.ConfigMap, name: oldap-yydccx-openldap-compdef-rsm-env, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.Service, name: oldap-yydccx-openldap-compdef-headless, immutable: false, orphan: false, action: UPDATE}->{obj:*v1.StatefulSet, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: UPDATE}->{obj:*v1alpha1.ReplicatedStateMachine, name: oldap-yydccx-openldap-compdef, immutable: false, orphan: false, action: STATUS} {"controller": "replicatedstatemachine", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "ReplicatedStateMachine", "ReplicatedStateMachine": {"name":"oldap-yydccx-openldap-compdef","namespace":"default"}, "namespace": "default", "name": "oldap-yydccx-openldap-compdef", "reconcileID": "0977a22f-8e47-4483-bbd4-bff940dc248e", "ReplicatedStateMachine": "default/oldap-yydccx-openldap-compdef"}
ahjing99 commented 1 year ago

➜ ~ kbcli report cluster oldap-yydccx --with-logs --all-containers reporting cluster information to report-cluster-oldap-yydccx-2023-09-22-14-01-03.zip processing manifests OK processing events OK process pod logs OK ➜ ~ kbcli report kubeblocks --with-logs --all-containers reporting KubeBlocks information to report-kubeblocks-2023-09-22-14-02-10.zip processing manifests OK processing events OK process pod logs OK report-kubeblocks-2023-09-22-14-02-10.zip report-cluster-oldap-yydccx-2023-09-22-14-01-03.zip

shanshanying commented 1 year ago

checked olap cluster, it's sts used emptyDir as volumes.

image
shanshanying commented 1 year ago

In oldap cd, it defines:

image

but in its cluster.yaml, it uses:

image

KubeBlocks will mapping volumes defined in ClusterDef with those in Cluster. If one intends to created PVC for his/her app, please make sure each volume is paired.

shanshanying commented 1 year ago

cc @kissycn