search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.24k stars 499 forks source link

operator tries to shrink volume after storage size update if value is in Mi #5651

Open mahdiG opened 6 months ago

mahdiG commented 6 months ago

Bug Report

What version of Kubernetes are you using? Client Version: v1.28.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.6

What version of TiDB Operator are you using? TiDB Operator Version: version.Info{GitVersion:"v1.5.3", GitCommit:"2c9e4dad0abaa4400afdef9ceff3084e71510ecb", GitTreeState:"clean", BuildDate:"2024-04-18T03:46:15Z", GoVersion:"go1.21.6", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods? cephfs.csi.ceph.com

What's the status of the TiDB cluster pods? Running

What did you do? I tried to update storage size in TidbCluster for PD and TiKV to 1740Mi and 15675Mi. The operator updated the volumes correctly but then in operator logs, I get these errors:

1 phase.go:74] volume rvgnrvpp/tikv-pnhrmptw-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
1 phase.go:74] volume rvgnrvpp/pd-pnhrmptw-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi

I set the disk size in Mi but after the volumes are updated, the operator seems to round up and use Gi and then complain about not being able to shrink the disk size from Gi value to Mi value that I've set.

What did you expect to see? I expected the operator NOT to try to shrink volumes after it updated the volumes.

What did you see instead? I saw the operator incorrectly assuming the storage size is 16Gi instead of 15675Mi and try to shrink the volumes.

csuzhangxc commented 6 months ago

Can you show the size of your PVCs (both TiKV and PD)?

mahdiG commented 6 months ago

The problem happens on create too, not just on update. I created a new TidbCluster and here's all the manifests and logs:

I created TidbCluster CR with the same storage sizes as before (15675Mi and 1740Mi). Here's the CR copied from Lens after creation:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  creationTimestamp: '2024-05-20T06:26:04Z'
  generation: 16
  labels:
    lexoya/component: '25'
    lexoya/name: wtiagtmv
    lexoya/version: '69'
  name: wtiagtmv
  namespace: rvgnrvpp
  resourceVersion: '572028224'
  uid: 77ce4dd6-fb5d-4814-9ca7-96393bec6d94
  selfLink: /apis/pingcap.com/v1alpha1/namespaces/rvgnrvpp/tidbclusters/wtiagtmv
status:
  clusterID: '7370964382394004606'
  conditions:
    - lastTransitionTime: '2024-05-20T06:28:23Z'
      lastUpdateTime: '2024-05-20T06:28:23Z'
      message: TiDB cluster is fully up and running
      reason: Ready
      status: 'True'
      type: Ready
  pd:
    image: repo.lexoya.com/pingcap/pd:v7.5.1
    leader:
      clientURL: http://wtiagtmv-pd-0.wtiagtmv-pd-peer.rvgnrvpp.svc:2379
      health: true
      id: '10249660128629638972'
      lastTransitionTime: '2024-05-20T06:26:27Z'
      name: wtiagtmv-pd-0
    members:
      wtiagtmv-pd-0:
        clientURL: http://wtiagtmv-pd-0.wtiagtmv-pd-peer.rvgnrvpp.svc:2379
        health: true
        id: '10249660128629638972'
        lastTransitionTime: '2024-05-20T06:26:27Z'
        name: wtiagtmv-pd-0
    phase: Normal
    statefulSet:
      collisionCount: 0
      currentReplicas: 1
      currentRevision: wtiagtmv-pd-6597dfcbd4
      observedGeneration: 1
      readyReplicas: 1
      replicas: 1
      updateRevision: wtiagtmv-pd-6597dfcbd4
      updatedReplicas: 1
    synced: true
    volumes:
      pd:
        boundCount: 1
        currentCapacity: 2Gi
        currentCount: 1
        currentStorageClass: csi-cephfs-sc-nvme
        modifiedCapacity: 1740Mi
        modifiedCount: 0
        modifiedStorageClass: csi-cephfs-sc-nvme
        name: pd
        resizedCapacity: 1740Mi
        resizedCount: 0
  pump: {}
  ticdc: {}
  tidb:
    image: repo.lexoya.com/pingcap/tidb:v7.5.1
    members:
      wtiagtmv-tidb-0:
        health: true
        lastTransitionTime: '2024-05-20T06:28:23Z'
        name: wtiagtmv-tidb-0
        node: 192.168.110.158
    phase: Normal
    statefulSet:
      collisionCount: 0
      currentReplicas: 1
      currentRevision: wtiagtmv-tidb-55f66f78b4
      observedGeneration: 1
      readyReplicas: 1
      replicas: 1
      updateRevision: wtiagtmv-tidb-55f66f78b4
      updatedReplicas: 1
  tiflash: {}
  tikv:
    bootStrapped: true
    conditions:
      - lastTransitionTime: '2024-05-20T06:27:13Z'
        message: Leader can be scheduled to all nodes
        reason: NoLeaderEviction
        status: 'False'
        type: LeaderEvicting
    image: repo.lexoya.com/pingcap/tikv:v7.5.1
    phase: Normal
    statefulSet:
      collisionCount: 0
      currentReplicas: 1
      currentRevision: wtiagtmv-tikv-5464f9546f
      observedGeneration: 1
      readyReplicas: 1
      replicas: 1
      updateRevision: wtiagtmv-tikv-5464f9546f
      updatedReplicas: 1
    stores:
      '1':
        id: '1'
        ip: wtiagtmv-tikv-0.wtiagtmv-tikv-peer.rvgnrvpp.svc
        lastTransitionTime: '2024-05-20T06:27:12Z'
        leaderCount: 60
        podName: wtiagtmv-tikv-0
        state: Up
    synced: true
    volumes:
      tikv:
        boundCount: 1
        currentCapacity: 16Gi
        currentCount: 1
        currentStorageClass: csi-cephfs-sc-nvme
        modifiedCapacity: 15675Mi
        modifiedCount: 0
        modifiedStorageClass: csi-cephfs-sc-nvme
        name: tikv
        resizedCapacity: 15675Mi
        resizedCount: 0
  tiproxy: {}
spec:
  configUpdateStrategy: RollingUpdate
  discovery:
    labels:
      lexoya/component: '25'
      lexoya/name: wtiagtmv
      lexoya/version: '69'
    limits:
      cpu: 50m
      memory: 100Mi
    requests:
      cpu: 1m
      memory: 20Mi
  enableDynamicConfiguration: true
  enablePVReclaim: false
  helper:
    image: repo.lexoya.com/library/alpine:3.16.0
  imagePullPolicy: IfNotPresent
  labels:
    lexoya/component: '25'
    lexoya/name: wtiagtmv
    lexoya/version: '69'
  pd:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/component
                    operator: In
                    values:
                      - pd
                  - key: lexoya/name
                    operator: In
                    values:
                      - wtiagtmv
              topologyKey: kubernetes.io/hostname
            weight: 1
    baseImage: repo.lexoya.com/pingcap/pd
    config: |
      [dashboard]
        internal-proxy = true

      [log]
        [log.file]
          max-backups = 1
          max-days = 1
          max-size = 100

      [replication]
        max-replicas = 1
    limits:
      cpu: '1.318'
      memory: 223M
    maxFailoverCount: 3
    mountClusterClientSecret: true
    replicas: 1
    requests:
      cpu: 180m
      memory: 120Mi
      storage: 1740Mi
    startTimeout: 30
  pvReclaimPolicy: Delete
  startScriptVersion: v2
  suspendAction: {}
  tidb:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/component
                    operator: In
                    values:
                      - tidb
                  - key: lexoya/name
                    operator: In
                    values:
                      - wtiagtmv
              topologyKey: kubernetes.io/hostname
            weight: 1
    baseImage: repo.lexoya.com/pingcap/tidb
    config: |
      [log]
        [log.file]
          max-backups = 1
          max-days = 1
          max-size = 100

      [performance]
        tcp-keep-alive = true
    limits:
      cpu: 467m
      memory: 1276M
    maxFailoverCount: 3
    replicas: 1
    requests:
      cpu: 100m
      memory: 200Mi
    service:
      type: ClusterIP
  tikv:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/component
                    operator: In
                    values:
                      - tikv
                  - key: lexoya/name
                    operator: In
                    values:
                      - wtiagtmv
              topologyKey: kubernetes.io/hostname
            weight: 1
    baseImage: repo.lexoya.com/pingcap/tikv
    config: |
      log-level = "info"

      [log]
        [log.file]
          max-backups = 1
          max-days = 1
          max-size = 100

      [raftdb]
        max-total-wal-size = "1GB"

      [rocksdb]
        max-total-wal-size = "1GB"

      [storage]
        reserve-space = "100MB"
    limits:
      cpu: 467m
      memory: 4511M
    maxFailoverCount: 3
    mountClusterClientSecret: true
    replicas: 1
    requests:
      cpu: 100m
      memory: 1536Mi
      storage: 15675Mi
    scalePolicy:
      scaleInParallelism: 1
      scaleOutParallelism: 1
  timezone: UTC
  tlsCluster: {}
  version: v7.5.1

TiKV pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tikv-wtiagtmv-tikv-0
  namespace: rvgnrvpp
  uid: 9d15043b-0636-45bb-9a1b-b1af1ab02948
  resourceVersion: '572027744'
  creationTimestamp: '2024-05-20T06:26:28Z'
  labels:
    app.kubernetes.io/component: tikv
    app.kubernetes.io/instance: wtiagtmv
    app.kubernetes.io/managed-by: tidb-operator
    app.kubernetes.io/name: tidb-cluster
    tidb.pingcap.com/cluster-id: '7370964382394004606'
    tidb.pingcap.com/pod-name: wtiagtmv-tikv-0
    tidb.pingcap.com/store-id: '1'
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    tidb.pingcap.com/pod-name: wtiagtmv-tikv-0
    volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/rvgnrvpp/persistentvolumeclaims/tikv-wtiagtmv-tikv-0
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 16Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15675Mi
  volumeName: pvc-9d15043b-0636-45bb-9a1b-b1af1ab02948
  storageClassName: csi-cephfs-sc-nvme
  volumeMode: Filesystem

PD pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pd-wtiagtmv-pd-0
  namespace: rvgnrvpp
  uid: ea461780-f79f-47b8-947b-f2701ce5d192
  resourceVersion: '572027731'
  creationTimestamp: '2024-05-20T06:26:06Z'
  labels:
    app.kubernetes.io/component: pd
    app.kubernetes.io/instance: wtiagtmv
    app.kubernetes.io/managed-by: tidb-operator
    app.kubernetes.io/name: tidb-cluster
    tidb.pingcap.com/cluster-id: '7370964382394004606'
    tidb.pingcap.com/member-id: '10249660128629638972'
    tidb.pingcap.com/pod-name: wtiagtmv-pd-0
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    tidb.pingcap.com/pod-name: wtiagtmv-pd-0
    volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/rvgnrvpp/persistentvolumeclaims/pd-wtiagtmv-pd-0
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 2Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1740Mi
  volumeName: pvc-ea461780-f79f-47b8-947b-f2701ce5d192
  storageClassName: csi-cephfs-sc-nvme
  volumeMode: Filesystem

operator logs:

E0520 06:26:05.033840       1 tidb_initializer_controller.go:106] TiDBInitializer: rvgnrvpp/wtiagtmv, sync failed, err: TidbInitManager.Sync: failed to get tidbcluster wtiagtmv for TidbInitializer rvgnrvpp/wtiagtmv, error: tidbcluster.pingcap.com "wtiagtmv" not found, requeuing
I0520 06:26:05.318603       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/wtiagtmv-tc-client-tls for controller TidbNGMonitoring/wtiagtmv successfully
I0520 06:26:05.318695       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.319347       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbDashboard", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"b39fbc16-cdb9-4a15-bf3a-c7a2e02f6d03", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026967", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb-dashboard-exposed in  wtiagtmv successful
I0520 06:26:05.353237       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.357424       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbDashboard", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"b39fbc16-cdb9-4a15-bf3a-c7a2e02f6d03", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026967", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tidb-dashboard in  wtiagtmv successful
I0520 06:26:05.365138       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-ng-monitoring in  wtiagtmv successful
I0520 06:26:05.393694       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-ng-monitoring-3961393 for controller TidbNGMonitoring/wtiagtmv successfully
I0520 06:26:05.417535       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-ng-monitoring in  wtiagtmv successful
I0520 06:26:05.512125       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create RoleBinding/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.544550       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Deployment/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.596824       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Service/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.859607       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-pd in  wtiagtmv successful
I0520 06:26:06.061752       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbInitializer", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"1fc1a34b-60d5-4c97-823c-6d2358e6378b", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026963", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tidb-initializer for controller TidbInitializer/wtiagtmv successfully
E0520 06:26:06.088485       1 tidb_initializer_controller.go:106] TiDBInitializer: rvgnrvpp/wtiagtmv, sync failed, err: updateStatus: failed to get job wtiagtmv-tidb-initializer for TidbInitializer rvgnrvpp/wtiagtmv, error: job.batch "wtiagtmv-tidb-initializer" not found, requeuing
I0520 06:26:06.088665       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbInitializer", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"1fc1a34b-60d5-4c97-823c-6d2358e6378b", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026963", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Job/wtiagtmv-tidb-initializer for controller TidbInitializer/wtiagtmv successfully
I0520 06:26:06.242077       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-pd-peer in  wtiagtmv successful
I0520 06:26:06.273267       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-pd-6334666 for controller TidbCluster/wtiagtmv successfully
I0520 06:26:06.301665       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-pd in  wtiagtmv successful
I0520 06:26:06.349515       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:06.349721       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for PD cluster running, requeuing
E0520 06:26:06.813751       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp: lookup wtiagtmv-pd.rvgnrvpp on 10.43.0.10:53: no such host, service rvgnrvpp/wtiagtmv-pd has no endpoints
I0520 06:26:06.943683       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:26:06.943885       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:07.348161       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp: lookup wtiagtmv-pd.rvgnrvpp on 10.43.0.10:53: no such host, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:07.387432       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:07.723528       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:07.761553       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
I0520 06:26:08.125232       1 tidb_init_manager.go:142] TidbInitializer: [rvgnrvpp/wtiagtmv] updated successfully
{"level":"warn","ts":"2024-05-20T06:26:10.220Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-23807697-7bf5-4485-93c1-93cb67f81e10/wtiagtmv-pd.rvgnrvpp:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.137.11:2379: connect: connection refused\""}
E0520 06:26:10.220875       1 monitor_manager.go:786] context deadline exceeded
E0520 06:26:10.220905       1 monitor_manager.go:107] Fail to sync TiDB Dashboard metrics config for TiDB cluster [rvgnrvpp/wtiagtmv], error: context deadline exceeded
I0520 06:26:10.230130       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/tidbmonitor-wtiagtmv-tls-assets for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.265579       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-prometheus in  wtiagtmv successful
I0520 06:26:10.300232       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-monitor-reloader in  wtiagtmv successful
I0520 06:26:10.373574       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-grafana in  wtiagtmv successful
I0520 06:26:10.400498       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.414949       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-monitor-grafana for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.431792       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.450916       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.463887       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.512205       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create RoleBinding/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
W0520 06:26:10.544937       1 warnings.go:70] spec.template.spec.initContainers[0].env[5].name: duplicate name "TZ"
I0520 06:26:10.545723       1 tidb_monitor_controller.go:99] TidbMonitor: rvgnrvpp/wtiagtmv, still need sync: TidbMonitor: [rvgnrvpp/wtiagtmv], waiting for tidbmonitor running, requeuing
I0520 06:26:10.545752       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-monitor in  wtiagtmv successful
I0520 06:26:10.545822       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Sync TidbMonitor[rvgnrvpp/wtiagtmv] Statefulset failed, err:TidbMonitor: [rvgnrvpp/wtiagtmv], waiting for tidbmonitor running
E0520 06:26:11.662535       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:11.692055       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:12.243252       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:12.282986       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:15.031799       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused
I0520 06:26:15.374715       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:26:15.374880       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
I0520 06:26:15.506073       1 tidb_dashboard_control.go:134] TidbDashboard: [rvgnrvpp/wtiagtmv], update status successfully
E0520 06:26:15.689451       1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused
E0520 06:26:15.729608       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
{"level":"warn","ts":"2024-05-20T06:26:16.546Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-23807697-7bf5-4485-93c1-93cb67f81e10/wtiagtmv-pd.rvgnrvpp:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.137.11:2379: connect: connection refused\""}
E0520 06:26:16.547071       1 monitor_manager.go:786] context deadline exceeded
E0520 06:26:16.547108       1 monitor_manager.go:107] Fail to sync TiDB Dashboard metrics config for TiDB cluster [rvgnrvpp/wtiagtmv], error: context deadline exceeded
I0520 06:26:16.859702       1 pv_control.go:192] PV: [pvc-e9d36715-8a9e-41d5-aec7-5cd142c33e46] updated successfully, : rvgnrvpp/wtiagtmv
I0520 06:26:16.893560       1 tidb_monitor_control.go:120] TidbMonitor: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:20.736268       1 tidb_ng_monitoring_control.go:153] TidbNGMonitoring: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:26:27.956642       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:28.002556       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tikv-peer in  wtiagtmv successful
I0520 06:26:28.032478       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tikv-6664313 for controller TidbCluster/wtiagtmv successfully
I0520 06:26:28.060629       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tikv in  wtiagtmv successful
I0520 06:26:28.121069       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:28.121242       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:28.624818       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:28.665325       1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:28.696456       1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:28.763600       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:28.763730       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:29.094536       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:29.135019       1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:29.187116       1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:29.188874       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:34.048099       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:34.112530       1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:34.149937       1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:34.213674       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:34.213802       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:34.648952       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:34.707919       1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:34.745607       1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:34.747280       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:42.170709       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:26:42.212448       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:26:42.310081       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:42.310189       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:42.596586       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:26:42.641823       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:26:42.686692       1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:27:12.215575       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:12.263871       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:12.325857       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb-peer in  wtiagtmv successful
I0520 06:27:12.375332       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb in  wtiagtmv successful
I0520 06:27:12.401228       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tidb-6436643 for controller TidbCluster/wtiagtmv successfully
I0520 06:27:12.433168       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tidb in  wtiagtmv successful
I0520 06:27:12.558263       1 pv_control.go:192] PV: [pvc-ea461780-f79f-47b8-947b-f2701ce5d192] updated successfully, : rvgnrvpp/wtiagtmv
I0520 06:27:12.926823       1 pv_control.go:192] PV: [pvc-9d15043b-0636-45bb-9a1b-b1af1ab02948] updated successfully, : rvgnrvpp/wtiagtmv
W0520 06:27:13.139695       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.140055       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:13.232600       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:27:13.232799       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed [modify volumes for rvgnrvpp/wtiagtmv:tidb failed: component phase is not Normal, modify volumes for rvgnrvpp/wtiagtmv:tikv failed: try to stop evicting leader for tidbcluster rvgnrvpp/wtiagtmv], requeuing
W0520 06:27:13.529859       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.574867       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:13.829302       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.829678       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:13.983249       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:27:13.983419       1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed modify volumes for rvgnrvpp/wtiagtmv:tikv failed: try to stop evicting leader for tidbcluster rvgnrvpp/wtiagtmv, requeuing
W0520 06:27:14.283262       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:14.326888       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:14.427504       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:14.427768       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:14.555305       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:15.029004       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.076644       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:15.281218       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.281579       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:15.605736       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.677335       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:16.352707       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:16.352959       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:42.231406       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.281047       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:42.442824       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.443135       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:42.528256       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:42.818005       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.921079       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:43.010650       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.010925       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:43.084599       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:43.300293       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.335028       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:43.850035       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.850247       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:02.520831       1 tidb_monitor_control.go:120] TidbMonitor: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:08.451478       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:08.501857       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:08.611842       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:08.612097       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:08.707193       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:09.222522       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:09.261375       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:09.349822       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:09.350090       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:12.245023       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.291253       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:12.398556       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.398755       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:12.544318       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:12.893828       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.953947       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:13.058740       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:13.058939       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.202208       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.242786       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.352464       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.352565       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:23.421251       1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:23.702868       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.743086       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.824304       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.824630       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:42.218587       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:42.266987       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:42.374751       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:42.374974       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:48.912223       1 tidb_init_manager.go:142] TidbInitializer: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:29:12.221467       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:12.273803       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:12.370491       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:12.370823       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:42.178022       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:42.220272       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:42.308498       1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:42.308846       1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi

You can ignore logs for other CRs. The TidbInitializer only creates an "app" database and does nothing else.

As you can see, operator creates the cluster with correct pvc sizes but then tries to shrink from a higher size to the original correct size.

csuzhangxc commented 6 months ago

from PVC

status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 16Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15675Mi

It seems the CSI driver created a 16Gi volume instead of a 15675Mi one.

mahdiG commented 6 months ago

from PVC

status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 16Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15675Mi

It seems the CSI driver created a 16Gi volume instead of a 15675Mi one.

Interesting. Maybe if operator looks at pvc spec.resources.requests.storage which is 15675Mi instead of pvc status.capacity.storage which is 16Gi, the problem is fixed. I'm not an expert but I think spec is a more solid thing to look at than status

csuzhangxc commented 6 months ago

from PVC

status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 16Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15675Mi

It seems the CSI driver created a 16Gi volume instead of a 15675Mi one.

Interesting. Maybe if operator looks at pvc spec.resources.requests.storage which is 15675Mi instead of pvc status.capacity.storage which is 16Gi, the problem is fixed. I'm not an expert but I think spec is a more solid thing to look at than status

When reszing a PVC, operator need to check the status to ensure whether the expanding operation is finished.