search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.68k stars 1.4k forks source link

restore with object metadata.ownerreference and metadata.finalizer #4707

Open polym opened 2 years ago

polym commented 2 years ago

Describe the problem/challenge you have [A description of the current limitation/problem/challenge that you are experiencing.]

In my scene, every deployment or statefulset will have a releated service if it is defined ports in podTemplate. When the deployment or statefulset is deleted, then the service related will be deleted by k8s gc as it has a owner reference to the deployemnt or statefulset.

I use velero to backup and restore for coping namespace from cluster a to cluster b, but I find the service in cluster b doesn't have ownerreference. After searching code, I find that may be releated to resetMetadataAndStatus https://github.com/vmware-tanzu/velero/blob/1e24d6ce718fc8fee890db34643e5be361fa5c7d/pkg/restore/restore.go#L1576 which delete all metadata other than name/namespace/labels/annotations.

Describe the solution you'd like [A clear and concise description of what you want to happen.]

restore with ownerreference and finalizer.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Issues or PRs may related:

  1. https://github.com/vmware-tanzu/velero/pull/837/files
  2. https://github.com/vmware-tanzu/velero/issues/2632

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

polym commented 2 years ago

Service example:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: deployment
    name: nginx-deployment
    uid: 13b53874-028e-4cc0-8a5f-ac413cb4bba9
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
reasonerjt commented 2 years ago

@polym Thanks for the issue. I think if we keep the ownerReferences in resetMetadataAndStatus your problem may be solved? I'm not sure if it's OK to replicate the uid field. This change was made years before I started to work on this project, so I think I need to discuss it with other maintainers regarding whether there's negative impacts if we decide to keep the ownerReferences during the restore.

polym commented 2 years ago

I'm not sure if it's OK to replicate the uid field.

@reasonerjt AFAIK, uid in the K8s cluster is unique, so, replicating the uid field will meet problems if copying ns1 to ns2 in the same K8s cluster.

New uid is fine in my scene. Now, my solution is this.

  1. Use velero restore plugin mechanism, to inject ownerreference/finializer which is encoded by json to annotation.
  2. Start a long-run program, to list K8s resources periodly, filter out the resources which contain ownerreference/finilizer annotation, get uid, and update the resources object metadata fields.

Otherwise, is it possible to control the order of creating resources, as uid in ownerreference is required. The solution will be more native If velero can control the order.

bsctl commented 2 years ago

@Lyndon-Li any progress on this?

Lyndon-Li commented 2 years ago

@bsctl This is a sophisticated problem for Velero, that is, Velero lacks a mechanism to handle resources' dependencies. We will do a comprehensive design, instead of just fixing the current problem. It takes time and also we need to find a proper release to take it in. So please keep tune on this issue, we will update once there is any progress.

Lyndon-Li commented 2 years ago

We will keep adding/linking the resource dependency related requirements/issues here, so that we have enough idea of how to make an ultimate solution.

Lyndon-Li commented 2 years ago

One more dependency case is the service's ExternalName feature: https://kubernetes.io/docs/concepts/services-networking/service/#externalname:~:text=pod%3A%20true-,Type%20ExternalName,-Services%20of%20type

By ExternalName, a service may be redirected to a target service in a different namespace. In a restore scenario, if the target service's namespace is changed, the source service will not work if the externalName field does not change.

Lyndon-Li commented 2 years ago

Another problem could be solved by a dependency solution: A sophisticated restore order could be defined according to resources' dependencies, so that there is no conflict for restore. An example here #5068