[BUG] mysql xtrabackup backup error: multipart upload failed to initialise

JashBook commented 2 hours ago

Describe the bug

kbcli version
Kubernetes: v1.28.3-vke.17
KubeBlocks: 0.9.2-beta.18
kbcli: 0.9.1

To Reproduce Steps to reproduce the behavior:

create backuprepo


kubectl get backuprepo
NAME                    STATUS   STORAGEPROVIDER   ACCESSMETHOD   DEFAULT   AGE
backuprepo-kbcli-test   Ready    minio             Tool           true      18d

kubectl describe backuprepo backuprepo-kbcli-test Name: backuprepo-kbcli-test Namespace:
Labels: Annotations: dataprotection.kubeblocks.io/backup-repo-digest: dccbce1192112ff2bb1e7c0593fb89cc dataprotection.kubeblocks.io/is-default-repo: true dataprotection.kubeblocks.io/need-update-tool-config: false API Version: dataprotection.kubeblocks.io/v1alpha1 Kind: BackupRepo Metadata: Creation Timestamp: 2024-10-28T01:31:36Z Finalizers: dataprotection.kubeblocks.io/finalizer Generation: 1 Resource Version: 8067102 UID: 086280d6-df0b-4fe8-98bf-f732bdc38aab Spec: Access Method: Tool Config: Bucket: kbcli-test Endpoint: http://kbcli-test-minio.kb-system.svc:9000 Geesefs Memory Limit: 512 Geesefs Read Ahead Large: 20480 Insecure: false Mount Options:
Region:
Credential: Name: kb-backuprepo-rt8lm Namespace: kb-system Pv Reclaim Policy: Retain Storage Provider Ref: minio Volume Capacity: 100Gi Status: Backup PVC Name: pvc-backuprepo-kbcli-test-5bczmt Conditions: Last Transition Time: 2024-10-28T01:31:36Z Message:
Observed Generation: 1 Reason: StorageProviderReady Status: True Type: StorageProviderReady Last Transition Time: 2024-10-28T01:31:36Z Message:
Observed Generation: 1 Reason: ParametersChecked Status: True Type: ParametersChecked Last Transition Time: 2024-10-28T01:31:36Z Message:
Observed Generation: 1 Reason: StorageClassCreated Status: True Type: StorageClassCreated Last Transition Time: 2024-10-28T01:31:36Z Message:
Observed Generation: 1 Reason: Skipped Status: True Type: PVCTemplateChecked Last Transition Time: 2024-10-28T01:31:45Z Message:
Observed Generation: 1 Reason: PreCheckPassed Status: True Type: PreCheckPassed Generated CSI Driver Secret: Name: secret-backuprepo-kbcli-test-w26fsn Namespace: kb-system Generated Storage Class Name: sc-backuprepo-kbcli-test-js5h7j Is Default: true Observed Generation: 1 Phase: Ready Tool Config Secret Name: tool-config-backuprepo-kbcli-test-k889z9 Events:

2. create mysql

apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: asmysql-yvujag namespace: default spec: clusterDefinitionRef: mysql topology: replication terminationPolicy: WipeOut componentSpecs:

name: mysql serviceVersion: 8.0.30 replicas: 2 resources: requests: cpu: 2000m memory: 4Gi limits: cpu: 2000m memory: 4Gi volumeClaimTemplates:
- name: data spec: storageClassName: accessModes:
  - ReadWriteOnce resources: requests: storage: 20Gi
```
3. load tpch data
```
    apiVersion: v1 kind: Pod metadata: name: tpchtest-asmysql-yvujag namespace: default spec: containers:
name: test-sysbench imagePullPolicy: IfNotPresent image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/customsuites:v0.0.1 env:
- name: TYPE value: "3"
- name: FLAG value: "7"
- name: CONFIGS value: "host:asmysql-yvujag-mysql.default.svc,db:tpchtest,user:root,password:Q41y5Q35Dv,port:3306,local:True,size:5"
restartPolicy: Never
```
4. xtrabackup
```
kbcli cluster backup asmysql-yvujag --method xtrabackup --namespace default
```
5. See error
```
kubectl logs dp-backup-0-backup-default-asmysql-yvujag-20241115122541-1cqbzr --namespace default --tail 500

Defaulted container "backupdata" out of: backupdata, manager, dp-copy-datasafed (init) 2024-11-15T04:26:17.847236-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql --datadir=/var/lib/mysql/data 2024-11-15T04:26:17.847306-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --backup=1 --safe-slave-backup=1 --slave-info=1 --stream=xbstream --host=asmysql-yvujag-mysql-0.asmysql-yvujag-mysql-headless --user=root --password=* xtrabackup version 8.0.35-31 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 55ec21d7) 2024-11-15T04:26:17.848186-00:00 0 [Note] [MY-011825] [Xtrabackup] perl binary not found. Skipping the version check 2024-11-15T04:26:17.848206-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: asmysql-yvujag-mysql-0.asmysql-yvujag-mysql-headless, user: root, password: set, port: not set, socket: not set 2024-11-15T04:26:18.097632-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.30 2024-11-15T04:30:23.611107-00:00 0 [Note] [MY-011825] [Xtrabackup] Slave open temp tables: 0 2024-11-15T04:30:23.613399-00:00 0 [Note] [MY-011825] [Xtrabackup] Slave is safe to backup. 2024-11-15T04:30:23.613480-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK INSTANCE FOR BACKUP ... 2024-11-15T04:30:23.633353-00:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise(). 2024-11-15T04:30:23.633387-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql/data 2024-11-15T04:30:23.633403-00:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 0, set to 1048576 2024-11-15T04:30:23.634118-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration: 2024-11-15T04:30:23.634131-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = . 2024-11-15T04:30:23.634137-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:12M:autoextend 2024-11-15T04:30:23.634168-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = ./ 2024-11-15T04:30:23.634173-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 2 2024-11-15T04:30:23.634180-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 50331648 2024-11-15T04:30:23.635690-00:00 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded 2024-11-15T04:30:23.674502-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: asmysql-yvujag-mysql-0.asmysql-yvujag-mysql-headless, user: root, password: set, port: not set, socket: not set 2024-11-15T04:30:23.706994-00:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up. 2024-11-15T04:30:23.776876-00:00 0 [Note] [MY-011825] [Xtrabackup] Starting to parse redo log at lsn = 8184153647 2024-11-15T04:30:23.777651-00:00 0 [Note] [MY-012564] [InnoDB] Recovery parsing buffer extended to 4194304. 2024-11-15T04:30:23.778365-00:00 0 [Note] [MY-012564] [InnoDB] Recovery parsing buffer extended to 8388608. 2024/11/15 04:30:24 NOTICE: S3 bucket kbcli-test: Streaming uploads using chunk size 50Mi will have maximum file size of 488.281Gi Error: push to "/backup-default-asmysql-yvujag-20241115122541.xbstream.zst": multipart upload failed to initialise: XMinioStorageFull: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. status code: 507, request id: 180809F9878D1D60, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 xtrabackup: Error writing file '' (OS errno 32 - Broken pipe) xb_stream_write_data() failed. 2024-11-15T04:30:24.080344-00:00 0 [ERROR] [MY-011825] [Xtrabackup] write to logfile failed failed with exit code 1



**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.

JashBook commented 2 hours ago

Increase minio pvc disk size

wangyelei commented 2 hours ago

volume expand the storage size for the minio

apecloud / kubeblocks

[BUG] mysql xtrabackup backup error: multipart upload failed to initialise #8462