kontena / pharos-cluster

Pharos - The Kubernetes Distribution
https://k8spharos.dev/
Apache License 2.0
312 stars 40 forks source link

Kontena-backup issue with persistent volumes #1052

Open suulperi opened 5 years ago

suulperi commented 5 years ago

For example namespace includes deployments or statefulsets with persistent volume claims. Backup will fail without PVC annonations backup will be success but off course volumes will not be backed up. Backups will be stored on minio server. Somehow log states restic repository is not ready even our repository is minio, not restic.

time="2019-02-05T12:37:37Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=kontena-backup/monitoring-with-pvc group=beta.kontena.io/v1 groupResource=clusters.beta.kontena.io logSource="pkg/backup/resource_backupper.go:154" time="2019-02-05T12:37:37Z" level=info msg="Backing up group" backup=kontena-backup/monitoring-with-pvc group=rook.io/v1alpha2 logSource="pkg/backup/group_backupper.go:142" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=rook.io/v1alpha2 groupResource=volumes.rook.io logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Listing items" backup=kontena-backup/monitoring-with-pvc group=rook.io/v1alpha2 groupResource=volumes.rook.io logSource="pkg/backup/resource_backupper.go:253" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Retrieved 0 items" backup=kontena-backup/monitoring-with-pvc group=rook.io/v1alpha2 groupResource=volumes.rook.io logSource="pkg/backup/resource_backupper.go:265" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Backing up group" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 logSource="pkg/backup/group_backupper.go:142" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=filesystems.ceph.rook.io logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Listing items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=filesystems.ceph.rook.io logSource="pkg/backup/resource_backupper.go:253" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Retrieved 0 items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=filesystems.ceph.rook.io logSource="pkg/backup/resource_backupper.go:265" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=objectstores.ceph.rook.io logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Listing items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=objectstores.ceph.rook.io logSource="pkg/backup/resource_backupper.go:253" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Retrieved 0 items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=objectstores.ceph.rook.io logSource="pkg/backup/resource_backupper.go:265" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=pools.ceph.rook.io logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Listing items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=pools.ceph.rook.io logSource="pkg/backup/resource_backupper.go:253" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Retrieved 0 items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=pools.ceph.rook.io logSource="pkg/backup/resource_backupper.go:265" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=clusters.ceph.rook.io logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Listing items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=clusters.ceph.rook.io logSource="pkg/backup/resource_backupper.go:253" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Retrieved 0 items" backup=kontena-backup/monitoring-with-pvc group=ceph.rook.io/v1beta1 groupResource=clusters.ceph.rook.io logSource="pkg/backup/resource_backupper.go:265" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Backing up group" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 logSource="pkg/backup/group_backupper.go:142" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=networkpolicies.extensions logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=kontena-backup/monitoring-with-pvc cohabitatingResource1=networkpolicies.extensions cohabitatingResource2=networkpolicies.networking.k8s.io group=extensions/v1beta1 groupResource=networkpolicies.extensions logSource="pkg/backup/resource_backupper.go:175" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=podsecuritypolicies.extensions logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=podsecuritypolicies.extensions logSource="pkg/backup/resource_backupper.go:154" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=ingresses.extensions logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Listing items" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=ingresses.extensions logSource="pkg/backup/resource_backupper.go:253" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Retrieved 3 items" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=ingresses.extensions logSource="pkg/backup/resource_backupper.go:265" namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Backing up resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=ingresses.extensions logSource="pkg/backup/item_backupper.go:177" name=grafana-ingress namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Backing up resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=ingresses.extensions logSource="pkg/backup/item_backupper.go:177" name=health-check namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Backing up resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=ingresses.extensions logSource="pkg/backup/item_backupper.go:177" name=prometheus namespace=monitoring time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=replicasets.extensions logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=kontena-backup/monitoring-with-pvc cohabitatingResource1=replicasets.extensions cohabitatingResource2=replicasets.apps group=extensions/v1beta1 groupResource=replicasets.extensions logSource="pkg/backup/resource_backupper.go:175" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=deployments.extensions logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=kontena-backup/monitoring-with-pvc cohabitatingResource1=deployments.extensions cohabitatingResource2=deployments.apps group=extensions/v1beta1 groupResource=deployments.extensions logSource="pkg/backup/resource_backupper.go:175" time="2019-02-05T12:37:37Z" level=info msg="Evaluating resource" backup=kontena-backup/monitoring-with-pvc group=extensions/v1beta1 groupResource=daemonsets.extensions logSource="pkg/backup/resource_backupper.go:138" time="2019-02-05T12:37:37Z" level=info msg="Skipping resource because it cohabitates and we've already processed it" backup=kontena-backup/monitoring-with-pvc cohabitatingResource1=daemonsets.extensions cohabitatingResource2=daemonsets.apps group=extensions/v1beta1 groupResource=daemonsets.extensions logSource="pkg/backup/resource_backupper.go:175" time="2019-02-05T12:37:37Z" level=info msg="Backup completed with errors: [restic repository is not ready, restic repository is not ready]" backup=kontena-backup/monitoring-with-pvc logSource="pkg/backup/backup.go:302"

jakolehm commented 5 years ago

As kontena-backup addon docs state, volume backups are stored to a secondary bucket (with -restic postfix). Do you have both buckets created?

Name of the bucket. Kontena Backup addon will also automatically create & use -restic bucket to store persistent volume backups