search

bitpoke / mysql-operator

Asynchronous MySQL Replication on Kubernetes using Percona Server and Openark's Orchestrator.
https://www.bitpoke.io/docs/mysql-operator/getting-started/
Apache License 2.0
993 stars 275 forks source link

Backups randomly fail #910

Open elderapo opened 8 months ago

elderapo commented 8 months ago

Recently started noticing that backups sometimes fail few times and then succeed:

Create Google Drive service-account.json file.
Create rclone.conf file.
I1106 00:00:02.894711       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="MY_SERVER_ID_OFFSET"
I1106 00:00:02.904695       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="MY_MYSQL_VERSION"
I1106 00:00:02.904702       1 deleg.go:130] sidecar "msg"="MY_MYSQL_VERSION is not a semver version"  
I1106 00:00:02.904707       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="MY_CLUSTER_NAME"
I1106 00:00:02.904712       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="MY_NAMESPACE"
I1106 00:00:02.904716       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="MY_SERVICE_NAME"
I1106 00:00:02.904720       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="INIT_BUCKET_URI"
I1106 00:00:02.904726       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="OPERATOR_USER"
I1106 00:00:02.904730       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="OPERATOR_PASSWORD"
I1106 00:00:02.904737       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="REPLICATION_USER"
I1106 00:00:02.904741       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="REPLICATION_PASSWORD"
I1106 00:00:02.904748       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="METRICS_EXPORTER_USER"
I1106 00:00:02.904752       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="METRICS_EXPORTER_PASSWORD"
I1106 00:00:02.904756       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="ORC_TOPOLOGY_USER"
I1106 00:00:02.904760       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="ORC_TOPOLOGY_PASSWORD"
I1106 00:00:02.904764       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="BACKUP_COMPRESS_COMMAND"
I1106 00:00:02.904767       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="BACKUP_DECOMPRESS_COMMAND"
I1106 00:00:02.904775       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="XBSTREAM_EXTRA_ARGS"
I1106 00:00:02.904785       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="XTRABACKUP_EXTRA_ARGS"
I1106 00:00:02.904789       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="XTRABACKUP_PREPARE_EXTRA_ARGS"
I1106 00:00:02.904795       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="XTRABACKUP_TARGET_DIR"
I1106 00:00:02.904799       1 deleg.go:130] sidecar "msg"="environment is not set"  "key"="INITFILE_EXTRA_SQL"
I1106 00:00:02.904847       1 deleg.go:130] sidecar "msg"="take a backup"  "bucket"="s3://*****/mysql-database-cluster-production/mysql-database-cluster-auto-2023-11-06t00-00-00.xbackup.gz" "host"="mysql-database-cluster-mysql-0.mysql.*****--production"
I1106 00:00:02.904858       1 deleg.go:130] sidecar "msg"="initialize a backup"  "endpoint"="/xbackup" "host"="mysql-database-cluster-mysql-0.mysql.*****--production"
2023/11/06 00:01:12 NOTICE: S3 bucket ***** path mysql-database-cluster-production: Streaming uploads using chunk size 10Mi will have maximum file size of 97.656Gi
I1106 00:44:26.740208       1 deleg.go:130] sidecar "msg"="backup was partially taken"  "trailers"={"X-Backup-Status":["Failed"]}
E1106 00:44:26.740257       1 deleg.go:144] sidecar "msg"="take backup command failed" "error"="backup failed to be taken: no 'Success' trailer found"
zerowebcorp commented 4 months ago

Getting the same issue. Backups are now continuously failing with that error message.