Researching Virtual Machine (kubevirt) migration with crane

migtools / crane

Tool for migrating Kubernetes workloads, and their data, between clusters.

https://www.konveyor.io/tools/crane/

Apache License 2.0

40 stars 23 forks source link

Researching Virtual Machine (kubevirt) migration with crane #91

Open djzager opened 2 years ago

djzager commented 2 years ago

This issue is a means of documenting the effort of migrating Virtual Machine's using crane.

djzager commented 2 years ago

The changes were made to the following projects as part of this effort:

crane - https://github.com/djzager/crane/tree/cnv-playground
cane-lib - https://github.com/djzager/crane-lib/tree/cnv-playground
cane-plugin-openshift - https://github.com/djzager/crane-plugin-openshift/tree/cnv-playground
crane-runner - https://github.com/djzager/crane-runner/tree/cnv-playground

Container Image: docker pull quay.io/djzager/crane-runner:cnv-playground .. this provides an image with all of the changes contained inside.

djzager commented 2 years ago

Issues that I've found

This section isn't complete until the issues listed are linked to actual GH Issues.

Crane's transfer-pvc needs to support specifying pod mutations.
Konveyor needs a solution for migrating PVCs with volumeMode: block
Konveyor needs a story for dealing with VirtualMachineSnapshots, and necessarily, VolumeSnapshots.
Crane export fails when you specify KUBECONFIG=/some/kubeconfig:/another/kubeconfig
We need a net new crane plugin...crane-plugin-kubevirt that can handle the mutations necessary for VirtualMachines. Further conversation needed on all the ways to handle this.

djzager commented 2 years ago

Environment Details

Using mig-agnosticd to provide two openshift 4.10 clusters (source cluster and destination cluster). The only thing that must be changed is the worker_instance_type must be m5.metal (related: OpenShift Virtualization on AWS).
Manually (via Console UI) installed the OpenShift Virtualization Operator and the HyperConverged CR in both source and destination clusters.
Mostly because writing Tekton Pipeline's is so much easier than having to write a shell script and just as portable (a file with .yaml or .sh), I also installed OpenShift Pipelines in the destination cluster.

djzager commented 2 years ago

Demonstration: https://youtu.be/tvDYvfoHNxg

djzager commented 2 years ago

PRs:

https://github.com/konveyor/crane-lib/pull/103 - updates crane-lib to drop default rbac and cabundles one would see in kube clusters, also cleans up managedFields from metadata as this isn't useful during migration
https://github.com/konveyor/crane-plugin-openshift/pull/11 - does similar things for openshift clusters

eriknelson commented 2 years ago

@djzager I think we need to split this one into tractable issues in GH as features to get them onto the priority roadmap and potentially turn this into an epic. I'll hold off on triaging this into Jira until we have a conversation about how that should be done.

eriknelson commented 2 years ago

/priority normal /kind feature