search

vmware-archive / kubernetes-archived

This repository is archived. Please file in-tree vSphere Cloud Provider issues at https://github.com/kubernetes/kubernetes/issues . CSI Driver for vSphere is available at https://github.com/kubernetes/cloud-provider-vsphere
Apache License 2.0
46 stars 31 forks source link

Backup of PVs not possible because of independent disk mode #302

Open embano1 opened 6 years ago

embano1 commented 6 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

Feature

What happened: It is not possible to backup an independent disk with most backup programs for vSphere, e.g. Veeam. The cloud provider creates PVs in mode "persistent independent". However, the opposite should be the case. Kubelet VMs should be of mode independent (nothing the cloud provider can do here), because they´re usually immutable deployments. PVs contain critical data (state) and as such must be backed up.

Disk creation mode is hard coded in the cloud provider: https://github.com/kubernetes/kubernetes/blob/7c13d65134f538a50f490ebb3567d08dd0841746/pkg/cloudprovider/providers/vsphere/vclib/virtualmachine.go#L268

What you expected to happen: Backup PVs with common vSphere backup software, e.g. Veeam

How to reproduce it (as minimally and precisely as possible): Use a backup software like Veeam and try to backup a VM with an attached PV.

Anything else we need to know?: What is the reason for making a disk of mode independent, besides excluding it from backup operations? What is the alternative way/ best practice of backing up PVs?

Environment:

BaluDontu commented 6 years ago

If we attach the disk as dependent disk to the VM, then any accidental snapshot of VM would create a snapshot of the disk. This means that the disk cannot be detached unless the VM snapshot is removed. Because of this the basic kubernetes workflows like pod rescheduling on other available nodes will not work since there are dependent snapshot disks which are still around.

So inorder to not run into any of these issues we have decided to attach disks as independent disks to VM.

jsafrane commented 6 years ago

Is there any way how to enable backups and keep the volume independent and safe from VM snapshots? It's quite natural that people want to backup their PVs.

JaredBurck commented 6 years ago

We are using CommVault for backups and hitting this same issue. Is there a workaround for being able to take snapshots of our PVs? Any updates or suggestions would be much appreciated!

cc: @eraid6

jsafrane commented 6 years ago

/sig vmware

@SandeepPissay, since you're assigned, do you have any insights how and when it could be fixed?

corrieb commented 6 years ago

Independent Persistent is necessary for the reasons highlighted above. You don't want the lifecycle of a PV tied to the lifecycle of the node. You particularly don't want the risk of a PV being deleted if a node is decommissioned. vSphere Integrated Containers uses the same VMDK mechanisms and I wrote some public documentation on the topic which may be of help here.

Top-level topic: https://vmware.github.io/vic-product/assets/files/html/1.3/vic_vsphere_admin/backup_vic.html

Specifics about backing up VMDK volumes: https://vmware.github.io/vic-product/assets/files/html/1.3/vic_vsphere_admin/backup_vmdk.html

Restoring VMDK volumes: https://vmware.github.io/vic-product/assets/files/html/1.3/vic_vsphere_admin/restore_vmdk.html

jsafrane commented 6 years ago

@corrieb (or anyone), can you please update Kubernetes docs to have this information?

corrieb commented 6 years ago

I can take a stab at summarizing what I wrote in the other docs. @jsafrane Do you have a sense of where it should go?

jsafrane commented 6 years ago

IMO, You should update https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/ - Kubernetes links it from https://kubernetes.io/docs/setup/pick-right-solution/

And then you can just link it from:

Source code for kubernetes.io/docs in in https://github.com/kubernetes/website, workflow is described at https://kubernetes.io/editdocs/

SandeepPissay commented 6 years ago

@jsafrane, @JaredBurck

VCP creates the PVs(Persistent Volumes) as independent disks as we do not want the PVs to be part of kubernetes node VM snapshot. There are complications if we do so. Also note that these disks cannot be managed independently.

However In vSphere 6.7 release, we can create and manage virtual storage objects independent of VM life cycle and this also has snapshot and backup support. The plan is to refactor the entire VCP code related to volume provisioning to start using virtual storage objects in vSphere. Once this is done, we should be able to backup the persistent volumes.

mkretzer commented 6 years ago

When will this refactoring be done? Will the plugin still be compatible with VSphere < 6.7 then?

SandeepPissay commented 6 years ago

@mkretzer : We are working on it and the plan is to get it out by end of this year. In terms of volume provisioning APIs in k8s, the plugin will be compatible with vSphere <= 6.7