oracle / mysql-operator

Create, operate and scale self-healing MySQL clusters in Kubernetes
870 stars 235 forks source link

Backups not excuting #292

Open umocku opened 5 years ago

umocku commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

MySQL Operator Version: iad.ocir.io/oracle/mysql-operator:0.3.0

Environment:

What happened?

Attempted to execute manual backup on cluster. The backup job gets scheduled and assigned to a mysql node, but fails upon execution.

What you expected to happen?

Backup to be scheduled and executed

How to reproduce it (as minimally and precisely as possible)?

Create mysql cluster with custom namespace on GKE, attempt to execute backup using the template in https://github.com/oracle/mysql-operator/blob/master/docs/user/backup.md

Anything else we need to know?

The error I see on the mysql-agent related to the pod where the mysql backup was scheduled is:

Could not construct reference to: '&v1alpha1.Backup{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"mysql-backup", GenerateName:"", Namespace:"lf-wordpress-cluster", SelfLink:"/apis/mysql.oracle.com/v1alpha1/namespaces/my-cluster/mysqlbackups/mysql-backup", UID:"3482f241-ese1-12e9-ac3b-52012a54120b", ResourceVersion:"6913943", Generation:3, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63706585526, loc:(*time.Location)(0x1d8c4a0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"v1alpha1.mysql.oracle.com/version":"0.3.0"}, Annotations:map[string]string{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"mysql.oracle.com/v1alpha1\",\"kind\":\"Backup\",\"metadata\":{\"annotations\":{},\"name\":\"mysql-backup\",\"namespace\":\"my-cluster\"},\"spec\":{\"cluster\":{\"name\":\"mysql\"},\"executor\":{\"mysqldump\":{\"databases\":[{\"name\":\"mydb\"}]}},\"storageProvider\":{\"s3\":{\"bucket\":\"mysql-backups\",\"credentialsSecret\":{\"name\":\"s3-credentials\"},\"endpoint\":\"storage.googleapis.com\",\"forcePathStyle\":true,\"region\":\"europe-west4\"}}}}\n"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1alpha1.BackupSpec{Executor:v1alpha1.BackupExecutor{MySQLDump:(*v1alpha1.MySQLDumpBackupExecutor)(0xc42025b080)}, StorageProvider:v1alpha1.StorageProvider{S3:(*v1alpha1.S3StorageProvider)(0xc42075f5c0)}, Cluster:(*v1.LocalObjectReference)(0xc4202d06a0), ScheduledMember:"mysql-2"}, Status:v1alpha1.BackupStatus{Outcome:v1alpha1.BackupOutcome{Location:""}, TimeStarted:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, TimeCompleted:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Conditions:[]v1alpha1.BackupCondition{v1alpha1.BackupCondition{Type:"Scheduled", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63706585526, loc:(*time.Location)(0x1d8c4a0)}}, Reason:"", Message:""}, v1alpha1.BackupCondition{Type:"Running", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63706585526, loc:(*time.Location)(0x1d8c4a0)}}, Reason:"", Message:""}}}}' due to: 'no kind is registered for the type v1alpha1.Backup'. Will not report event: 'Warning' 'ExecutionFailed' 'mkdir /var/lib/mysql-agent/backup: permission denied'

There seems to be two messages in there: one that 'mkdir /var/lib/mysql-agent/backup fails with permission denied. Upon inspection, I can see that the folder /var/lib/mysql-agent/ has chown root:root, but the mysql-agent runs with user mysql. I do not know what the Could not construct reference error refers to or if it is related to the mkdir failing.