search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.23k stars 498 forks source link

Backup to Alibaba OSS error（dumpling）：Rclone S3 excess default ACL exists #4769

Open jangrui opened 1 year ago

jangrui commented 1 year ago

Deployment environment

bk

cat <<-EOF | kubectl apply -f -
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
  name: gpsp
  namespace: tidb-cluster
spec:
  from:
    host: gpsp-tidb
    port: 4000
    user: root
    secretName: gpsp-tidb-backup-secret
  s3:
    provider: alibaba
    region: oss-cn-zhangjiakou
    bucket: gpsp
    prefix: tidb-backup
    endpoint: https://gpsp.oss-cn-zhangjiakou.aliyuncs.com
    secretName: oss-secret
  dumpling:
    options:
      - --threads=16
      - --rows=10000
  storageClassName: longhorn
  storageSize: 100Gi
EOF

error logs

Create rclone.conf file.
/tidb-backup-manager export --namespace=tidb-cluster --backupName=gpsp --bucket=gpsp --storageType=s3
I1106 01:58:46.110902       9 export.go:74] start to process backup tidb-cluster/gpsp
I1106 01:58:46.152348       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
I1106 01:58:46.160026       9 manager.go:179] cluster tidb-cluster/gpsp tikv_gc_life_time is 10m0s
I1106 01:58:46.173253       9 manager.go:243] set cluster tidb-cluster/gpsp tikv_gc_life_time to 72h success
E1106 01:58:46.184620       9 backup_status_updater.go:89] Failed to update backup [tidb-cluster/gpsp], error: Operation cannot be fulfilled on backups.pingcap.com "gpsp": the object has been modified; please apply your changes to the latest version and try again
I1106 01:58:46.209279       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
I1106 01:58:46.219249       9 export.go:97] The dump process is ready, command "/dumpling --output=/backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z --host=gpsp-tidb --port=4000 --user=root --password=****** --filter *.* --filter !/^(mysql|test|INFORMATION_SCHEMA|PERFORMANCE_SCHEMA|METRICS_SCHEMA|INSPECTION_SCHEMA)$/.* --threads=16 --rows=10000"
I1106 02:00:19.057815       9 manager.go:284] reset cluster tidb-cluster/gpsp tikv_gc_life_time to 10m0s success
I1106 02:00:19.057853       9 manager.go:299] dump cluster tidb-cluster/gpsp data to /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z success
I1106 02:00:19.058051       9 manager.go:314] get cluster tidb-cluster/gpsp commitTs 437169979755855873 success
I1106 02:05:14.457187       9 manager.go:329] archive cluster tidb-cluster/gpsp backup data /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz success
I1106 02:05:14.487832       9 manager.go:345] get cluster tidb-cluster/gpsp archived backup file /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz size 1120088963 success
I1106 02:05:44.599196       9 export.go:116] cluster tidb-cluster/gpsp, rclone copy data from /backup/gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz to s3:gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz.tmp, log: 2022/11/06 02:05:18 INFO  : S3 bucket gpsp path tidb-backup: Bucket "gpsp" created with ACL "private"
2022/11/06 02:05:44 INFO  : backup-2022-11-05T17:58:46Z.tgz: Copied (new)
2022/11/06 02:05:44 INFO  :
Transferred:        1.043G / 1.043 GBytes, 100%, 40.310 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        26.4s

I1106 02:05:44.599258       9 export.go:117] upload cluster tidb-cluster/gpsp backup data to s3:gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz.tmp successfully, now move it to permanent URL s3:gpsp/tidb-backup/backup-2022-11-05T17:58:46Z.tgz
E1106 02:05:45.470609       9 manager.go:353] backup cluster tidb-cluster/gpsp data to s3 failed, err: cluster tidb-cluster/gpsp, execute rclone moveto command failed, output: 2022/11/06 02:05:45 INFO  : S3 bucket gpsp path tidb-backup: Bucket "gpsp" created with ACL "private"
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : Attempt 1/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : Attempt 2/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : Attempt 3/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 Failed to moveto: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
, err: exit status 1
E1106 02:05:45.482902       9 backup_status_updater.go:89] Failed to update backup [tidb-cluster/gpsp], error: Operation cannot be fulfilled on backups.pingcap.com "gpsp": the object has been modified; please apply your changes to the latest version and try again
I1106 02:05:45.516995       9 backup_status_updater.go:86] Backup: [tidb-cluster/gpsp] updated successfully
error: cluster tidb-cluster/gpsp, execute rclone moveto command failed, output: 2022/11/06 02:05:45 INFO  : S3 bucket gpsp path tidb-backup: Bucket "gpsp" created with ACL "private"
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : Attempt 1/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC4936319EC481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : Attempt 2/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC493631E2C481, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Failed to copy: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : backup-2022-11-05T17:58:46Z.tgz.tmp: Not deleting source as copy failed: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 ERROR : Attempt 3/3 failed with 1 errors and: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
2022/11/06 02:05:45 Failed to moveto: NoSuchKey: The specified key does not exist.
    status code: 404, request id: 6366A5F942FC49363107C581, host id:
, err: exit status 1
jangrui commented 1 year ago

https://github.com/rclone/rclone/pull/5731