search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.45k stars 1.37k forks source link

All cluster-scoped resources are restored when selecting a specific namespace during restore #1914

Closed adityagu0910 closed 4 years ago

adityagu0910 commented 4 years ago

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.)

I am taking nightly full cluster backup(that includes all Namespaces in my k8s cluster) and restoring one namespace. Restore looks good and i am bale to access my application but I see one warning in my restore which contains a resource which is not part of my restore namespace. Could you please suggest why we are getting this warning and why it is trying to restore this resource when i am not even including that resource related namespace

velero restore describe restore-one-namespace-lrc-mnrf-fsims-dev --details Name: restore-one-namespace-lrc-mnrf-fsims-dev Namespace: velero Labels: Annotations:

Phase: Completed

Warnings: Velero: *_Cluster: not restored: persistentvolumes "logging-datanode-" already exists and is different from backed up version._** Namespaces:

Backup: whole-cluster-backup-26092019112127

Namespaces: Included: lrc-mnrf-fsims-dev Excluded:

Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io Cluster-scoped: auto

Namespace mappings:

Label selector:

Restore PVs: auto

Restic Restores: Completed: lrc-mnrf-fsims-dev/lrc-mnrf-fsims-dev-relea-0: liberty-pvc

What did you expect to happen:

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other pastebin is fine.)

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

adityagu0910 commented 4 years ago

I am getting this warning messages because I stopped taking backup of this PV as it was not required.

time="2019-09-26T19:18:55Z" level=info msg="No snapshot found for persistent volume" logSource="pkg/restore/pv_restorer.go:81" persistentVolume=logging-datanode-IP restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:933" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:63" pluginName=velero restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=debug msg="Getting plugin config" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:66" pluginName=velero restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=debug msg="No storage class mappings found" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:73" pluginName=velero restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:74" pluginName=velero restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore PersistentVolume: logging-datanode-IP" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev

I noticed that velero was trying to restore "CLUSTER LEVEL RESOURCES" and I think it should not try to restore cluster level resource when I am trying to restore only one namespace.

time="2019-09-26T19:18:55Z" level=info msg="Restoring cluster level resource 'storageclasses.storage.k8s.io' from: /tmp/286041513/resources/storageclasses.storage.k8s.io/cluster" logSource="pkg/restore/restore.go:726" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Getting client for storage.k8s.io/v1, Kind=StorageClass" logSource="pkg/restore/restore.go:772" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: ceph-storage" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: ceph-storage because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev

I see there is flag for that and after adding flag --include-cluster-resources=false I see it is not trying to restore cluster level resource. But my restore is stuck on restic pv restore and it shows its state as New. Phase shows as "InProgress" and I do not see any error in velero pod log.

velero restore describe restore-one-namespace-tests-lrc-mnrf-fsims-dev --details Name: restore-one-namespace-tests-lrc-mnrf-fsims-dev Namespace: velero Labels: Annotations:

Phase: InProgress

Backup: whole-cluster-backup-26092019112127

Namespaces: Included: lrc-mnrf-fsims-dev Excluded:

Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io Cluster-scoped: excluded

Namespace mappings:

Label selector:

Restore PVs: auto

Restic Restores: New: lrc-mnrf-fsims-dev/lrc-mnrf-fsims-dev-relea-0: liberty-pvc

adityagu0910 commented 4 years ago

I see below resource status in namespace - I believe PersistentVolume are cluster level resource so it is not able to restore corresponding PVC.

kubectl get all
NAME                             READY   STATUS    RESTARTS   AGE
pod/lrc-mnrf-fsims-dev-relea-0   0/2     Pending   0          38m

NAME                                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/glusterfs-dynamic-3fe005f8-e066-11e9-86fd-0050568abba1   ClusterIP   172.16.132.150   <none>        1/TCP      38m
service/lrc-mnrf-fsims-gocloud-service                           ClusterIP   172.16.130.201   <none>        9443/TCP   38m
service/lrc-mnrf-fsims-gocloud-service-sts                       ClusterIP   None             <none>        9443/TCP   38m

NAME                                        READY   AGE
statefulset.apps/lrc-mnrf-fsims-dev-relea   0/1     38m
[root@IP ~]# kubectl get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
liberty-pvc-lrc-mnrf-fsims-dev-relea-0   Lost     pvc-3fe005f8-e066-11e9-86fd-0050568abba1   0                         glusterfs      38m

[root@IP~]# kubectl get events
LAST SEEN   TYPE      REASON             KIND                    MESSAGE
43m         Warning   ClaimLost          PersistentVolumeClaim   Bound claim has lost its PersistentVolume. Data on the volume is lost!
2m57s       Warning   FailedScheduling   Pod                     could not find v1.PersistentVolume "pvc-3fe005f8-e066-11e9-86fd-0050568abba1" (repeated 2 times)
43m         Normal    CREATE             Ingress                 Ingress lrc-mnrf-fsims-dev/lrc-mnrf-fsims-dev-relea
42m         Normal    UPDATE             Ingress                 Ingress lrc-mnrf-fsims-dev/lrc-mnrf-fsims-dev-relea
adityagu0910 commented 4 years ago

restore completed with below message

time="2019-09-27T16:49:07Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="timed out waiting for all PodVolumeRestores to complete" logSource="pkg/restore/restore.go:1126" restore=velero/restore-one-namespace-tests-lrc-mnrf-fsims-dev

I am afraid of cluster level resources getting restore which we do not want when we are restoring only one namespace.

nrb commented 4 years ago

@adityagu0910 This appears to be intended behavior.

If a volume is selected for backup on a pod, Velero will also capture the related PV based on the assumption that the Pod is not useful without the PV. If you don't include the PV, you'll see the behavior you describe - the PVC references a PV that doesn't exist.

Are you observing any other cluster-level resources being recreated, besides the PV for liberty-pvc-lrc-mnrf-fsims-dev-relea-0? What would you expect Velero to do if it didn't recreate the PV?

adityagu0910 commented 4 years ago

Thanks for reply Nolan

It should recreate PV that is related to namespace which I am trying to restore. (expected behavior)

But it seems to run restore for other cluster level resources as well although it says "Skipping restore of "resource name" because it already exists in the cluster and is unchanged from the backed up version"

Below are some snippets from log of restore -

level=info msg="Starting restore of backup velero/whole-cluster-backup-26092019112127" logSource="pkg/restore/restore.go:377" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Restoring cluster level resource 'storageclasses.storage.k8s.io' from: /tmp/286041513/resources/storageclasses.storage.k8s.io/cluster" logSource="pkg/restore/restore.go:726" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Getting client for storage.k8s.io/v1, Kind=StorageClass" logSource="pkg/restore/restore.go:772" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: ceph-storage" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: ceph-storage because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: glusterfs" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: glusterfs because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: image-manager-storage" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: image-manager-storage because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: kafka-storage" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: kafka-storage because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: logging-storage-datanode" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: logging-storage-datanode because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: minio-storage" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: minio-storage because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore StorageClass: mongodb-storage" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of StorageClass: mongodb-storage because it already exi

time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore CustomResourceDefinition: alertrules.monitoringcontroller.cloud.ibm.com" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of CustomResourceDefinition: alertrules.monitoringcontroller.cloud.ibm.com because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore CustomResourceDefinition: backups.velero.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of CustomResourceDefinition: backups.velero.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore CustomResourceDefinition: backupstoragelocations.velero.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Skipping restore of CustomResourceDefinition: backupstoragelocations.velero.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:55Z" level=info msg="Attempting to restore CustomResourceDefinition: certificates.certmanager.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: certificates.certmanager.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition: challenges.certmanager.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: challenges.certmanager.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition: clients.oidc.security.ibm.com" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: clients.oidc.security.ibm.com because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition: clusterimagepolicies.securityenforcement.admission.cloud.ibm.com" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: clusterimagepolicies.securityenforcement.admission.cloud.ibm.com because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition: clusterissuers.certmanager.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: clusterissuers.certmanager.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition: clusters.ceph.rook.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: clusters.ceph.rook.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition: deletebackuprequests.velero.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Skipping restore of CustomResourceDefinition: deletebackuprequests.velero.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:56Z" level=info msg="Attempting to restore CustomResourceDefinition:

time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.icp.ibm.com because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1.monitoringcontroller.cloud.ibm.com" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.monitoringcontroller.cloud.ibm.com because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1.networking.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.networking.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1.oidc.security.ibm.com" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.oidc.security.ibm.com because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1.rbac.authorization.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.rbac.authorization.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1.storage.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.storage.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1.velero.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1.velero.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1alpha1.admissionregistration.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Skipping restore of APIService: v1alpha1.admissionregistration.k8s.io because it already exists in the cluster and is unchanged from the backed up version" logSource="pkg/restore/restore.go:1088" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:18:59Z" level=info msg="Attempting to restore APIService: v1alpha1.certmanager.k8s.io" logSource="pkg/restore/restore.go:1031" restore=velero/restore-one-namespace-lrc-mnrf-fsims-dev time="2019-09-26T19:19:00Z" level=info msg="Skipping restore of APIService: v1alpha1.

I am concerned about these messages for below scenario. One day someone deleted some cluster level resources those are not required and when next time we run restore for one namespace only, it will restore those deleted cluster level resources as well.

I have not tested this yet but I will run this test and will update you with result if it tries to restore any such cluster level resources.

nrb commented 4 years ago

Can you provide the output of the following commands?

velero get backup whole-cluster-backup-26092019112127 -o yaml velero get restore restore-one-namespace-tests-lrc-mnrf-fsims-dev -o yaml

Also, let me summarize the problem to be sure I understand:

You've taken a backup of the entire cluster. You're trying to selectively restore a single namespace, which has at least 1 PV. However, upon restore, PVs that are not associated with the PV are also being processed, though not actually added (per the warning in the logs). These PVs could be a problem in the future because they could be restored from a backup made prior to their intentional deletion.

Is that an accurate summary?

adityagu0910 commented 4 years ago

Yeah that is correct but not only for PV as it could be with any other cluster level resource. (like StorageClass, clusterrolebinding, ClusterRole, ImagePolicy)

Please see below requested o/ps

velero get backup whole-cluster-backup-26092019112127 -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  creationTimestamp: 2019-09-26T15:22:26Z
  generation: 3
  labels:
    velero.io/storage-location: default
  name: whole-cluster-backup-26092019112127
  namespace: velero
  resourceVersion: "12789939"
  selfLink: /apis/velero.io/v1/namespaces/velero/backups/whole-cluster-backup-26092019112127
  uid: 727c1083-e071-11e9-86fd-0050568abba1
spec:
  excludedNamespaces: null
  excludedResources: null
  hooks:
    resources: null
  includeClusterResources: null
  includedNamespaces:
  - '*'
  includedResources: null
  labelSelector: null
  storageLocation: default
  ttl: 720h0m0s
  volumeSnapshotLocations: null
status:
  completionTimestamp: 2019-09-26T15:32:21Z
  errors: 0
  expiration: 2019-10-26T15:22:26Z
  phase: Completed
  startTimestamp: 2019-09-26T15:22:26Z
  validationErrors: null
  version: 1
  volumeSnapshotsAttempted: 0
  volumeSnapshotsCompleted: 0
  warnings: 0
velero get restore restore-one-namespace-tests-lrc-mnrf-fsims-dev -o yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: 2019-09-27T15:49:07Z
  generation: 3
  name: restore-one-namespace-tests-lrc-mnrf-fsims-dev
  namespace: velero
  resourceVersion: "13029290"
  selfLink: /apis/velero.io/v1/namespaces/velero/restores/restore-one-namespace-tests-lrc-mnrf-fsims-dev
  uid: 56c97f7f-e13e-11e9-86fd-0050568abba1
spec:
  backupName: whole-cluster-backup-26092019112127
  excludedNamespaces: null
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  includeClusterResources: false
  includedNamespaces:
  - lrc-mnrf-fsims-dev
  includedResources: null
  namespaceMapping: {}
status:
  errors: 0
  failureReason: ""
  phase: Completed
  validationErrors: null
  warnings: 0
velero get restore restore-one-namespace-test1-lrc-mnrf-fsims-dev -o yaml 
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: 2019-09-27T15:43:11Z
  generation: 3
  name: restore-one-namespace-test1-lrc-mnrf-fsims-dev
  namespace: velero
  resourceVersion: "13019217"
  selfLink: /apis/velero.io/v1/namespaces/velero/restores/restore-one-namespace-test1-lrc-mnrf-fsims-dev
  uid: 82f928d4-e13d-11e9-86fd-0050568abba1
spec:
  backupName: whole-cluster-backup-26092019112127
  excludedNamespaces: null
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  includedNamespaces:
  - lrc-mnrf-fsims-dev
  includedResources: null
  namespaceMapping: {}
status:
  errors: 0
  failureReason: ""
  phase: Completed
  validationErrors: null
  warnings: 1
nrb commented 4 years ago

Thanks, this is clearer now.

I think we may need to adjust our selective restore logic when it comes to including cluster resources. By default, I believe what we're doing is actually restoring the specified namespace and its contents plus all other cluster-scoped resources. However, by excluding all cluster-scoped resources, we lose any PVs that should be included.

I'll have to think about how we might make this behave closer to the backup code, where we more accurately only include referenced cluster-scoped resources (such as PVs) on restore.

@skriss Can you think of a work around for this at the moment? Other than making smaller scoped backups, which can be a pain, we might have to review our selective restore.

skriss commented 4 years ago

Hmm, yeah, it does look like the restore logic here is inconsistent with backups. I'd consider this a bug.

Yeah, the best workaround I can think of at the moment is doing a backup per namespace, and restoring that.

adityagu0910 commented 4 years ago

Thanks Nolan and Steve !! I will use namespace level backup/restore for now.

keerthi0489 commented 4 years ago

I am having the exact same issue as well, sadly for our use case we need to take the backup of the entire cluster and may need to restore namespace level resources alone + corresponding PVs

skriss commented 4 years ago

we'd like to fix this in v1.3. If anyone is interested in working on the fix, let us know!

ssethuma1155 commented 3 years ago

I am running into this issue with velero 1.5.2 on OpenShift. Do you know what version this fix will be available?

carlisia commented 3 years ago

Yes, v1.6, to be release end of Jan/beginning of Feb.

ssethuma1155 commented 3 years ago

Great. Thank you

ssethuma1155 commented 3 years ago

Is v1.6 released yet? and is the fix for this issue in v1.6? Thank you.

ssethuma1155 commented 3 years ago

I download 1.6.rc1 and unfortunately this does not seem to have been fixed. When trying to set includeClusterResources: nil (or null) in the restore yaml file, I see an error 'incorrect type. Expected: boolean'. Does not work from command line either. Possible that I am not doing this right. Would appreciate some help.

DevFontes commented 2 years ago

I am using client version 1.7.2 and server 1.8.0 and still have this problem.

VGerris commented 1 year ago

also seems to happen with a full backup and restore with –include-cluster-resources=false .