Unable to Find MySQL Backups in S3 Bucket Despite Successful Pod Status

[Suraj-Waghule]

Description:

Issue Overview: Despite successful completion status of backup pods, MySQL backups are not found in the specified S3 bucket.

Expected Behavior: Backups of MySQL databases should be successfully uploaded to the designated S3 bucket upon completion of backup pods.

Actual Behavior: Although backup pods indicate completion status, the backups are not present in the specified S3 bucket.

Steps to Reproduce:

Configure backup process to take regular backups of MySQL databases using IRSA. Create and attach necessary policies to the role for IRSA. Verify pod status using command kubectl get pods. Check the S3 bucket for the presence of MySQL backups.

Additional Information:

MySQL Operator Pod is running without any issues. All backup pods show a status of "Completed". The IAM role for IRSA has been created and attached with necessary policies. Despite successful completion of backup pods, no backups are found in the S3 bucket.

Attachments:

[2024-06-10 12:03:49,150] kopf.objects         [INFO    ] on_innodbcluster_field_backup_schedules
[2024-06-10 12:03:49,152] kopf.objects         [INFO    ] backup_objects.updates_schedules
[2024-06-10 12:03:49,153] kopf.objects         [INFO    ] backup_objects.compare_schedules mysql-operator/mycluster
[2024-06-10 12:03:49,154] kopf.objects         [INFO    ] backup_objects.update_schedules: diff={'removed': {}, 'added': {}, 'modified': {'s3-backup-schedule': {'old': <mysqloperator.controller.backup.backup_api.BackupSchedule object at 0x7fb8639bb7f0>, 'new': <mysqloperator.controller.backup.backup_api.BackupSchedule object at 0x7fb863c8e550>}}, 'unmodified': {}}
[2024-06-10 12:03:49,155] kopf.objects         [INFO    ] backup_objects.update_schedules: will modify 1 backup schedule objects
[2024-06-10 12:03:49,166] kopf.objects         [INFO    ] backup_objects.update_schedules: modifying schedule mycluster-s3-backup-schedule-cb in mysql-operator
[2024-06-10 12:03:49,169] kopf.objects         [INFO    ] backup_objects.update_schedules: {'apiVersion': 'batch/v1', 'kind': 'CronJob', 'metadata': {'labels': {'tier': 'mysql', 'mysql.oracle.com/cluster': 'mycluster', 'app.kubernetes.io/name': 'mysql-innodbcluster', 'app.kubernetes.io/instance': 'idc-mycluster', 'app.kubernetes.io/managed-by': 'mysql-operator', 'app.kubernetes.io/created-by': 'mysql-operator'}, 'name': 'mycluster-s3-backup-schedule-cb'}, 'spec': {'concurrencyPolicy': 'Forbid', 'jobTemplate': {'spec': {'backoffLimit': 0, 'template': {'spec': {'securityContext': {'runAsUser': 27, 'runAsGroup': 27, 'fsGroup': 27}, 'containers': [{'name': 'operator-backup-job-cron', 'image': 'container-registry.oracle.com/mysql/community-operator:8.4.0-2.1.3', 'imagePullPolicy': 'IfNotPresent', 'command': ['mysqlsh', '--pym', 'mysqloperator', 'backup', '--command', 'create-backup-object', '--namespace', 'mysql-operator', '--cluster-name', 'mycluster', '--schedule-name', 's3-backup-schedule'], 'securityContext': {'allowPrivilegeEscalation': False, 'privileged': False, 'readOnlyRootFilesystem': True, 'capabilities': {'drop': ['ALL']}}, 'env': [{'name': 'MYSQLSH_USER_CONFIG_HOME', 'value': '/mysqlsh'}], 'volumeMounts': [{'name': 'shellhome', 'mountPath': '/mysqlsh'}]}], 'volumes': [{'name': 'shellhome', 'emptyDir': {}}], 'restartPolicy': 'Never', 'terminationGracePeriodSeconds': 60, 'serviceAccountName': 'mycluster-sa'}}}}, 'suspend': False, 'schedule': '0 0 * * *', 'timeZone': 'Asia/Kolkata'}}
[2024-06-10 12:03:49,191] kopf.objects         [INFO    ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' succeeded.
[2024-06-10 12:03:49,191] kopf.objects         [INFO    ] Updating is processed: 1 succeeded; 0 failed.

[github-actions[bot]]

https://warthogs.atlassian.net/browse/DPE-4591

[taurus-forever]

Dear @Suraj-Waghule can you please share the name of charm you deployed? Are you testing https://charmhub.io/mysql-k8s (we are in right repo then)?

The Attachments provided looks new to me, maybe wrong repo reported? Please share the deployed charm name / channel and it's revision. Thank you!

[paulomach]

Closing as invalid issue.