search

kubernetes / cloud-provider

cloud-provider defines the shared interfaces which Kubernetes cloud providers implement. These interfaces allow various controllers to integrate with any cloud provider in a pluggable fashion. Also serves as an issue tracker for SIG Cloud Provider.
Apache License 2.0
245 stars 110 forks source link

Document migration steps to CCM #42

Open andrewsykim opened 4 years ago

andrewsykim commented 4 years ago

We should document how a user would manually migrate their clusters from using in-tree cloud providers to out-of-tree cloud provider. The documented steps can be manual or via a tool like kubeadm.

onitake commented 4 years ago

To get started, a rough outline:

  1. Ensure a CCM for your cloud environment is available, with roughly the same feature set than the integrated KCM provider. Determine compatibility issues (missing features, different implementation, etc).
  2. Prepare your cloud environment and workloads for the migration: Disable unsupported features, build a list of manual actions to be done after migration (deleting unused cloud resources, renaming, etc).
  3. Prepare the CCM for deployment: Write configuration files, deploy credentials, etc. Do not deploy the CCM yet
  4. Disable the integrated provider in KCM and kubelet: Remove flags, replace with --provider external, etc. Restart these services.
  5. Deploy the cloud provider.
  6. Ensure it synchronises with the running environment correctly, detects existing resources, deploys new resources where appropriate. Apply manual fixes where necessary.
  7. Test deploy new cloud resources such as LoadBalancers and Nodes.

There will be certain differences between different cloud providers, as compatibility between integrated and external cannot always be guaranteed.

andrewsykim commented 4 years ago

I think this warrants a page in the official Kubernetes docs, @onitake are you willing to put something together?

onitake commented 4 years ago

Yes, I think I can do that. But I will need more input, and possibly some insight on the situation with different providers.

onitake commented 4 years ago

And also, where should the documentation live?

andrewsykim commented 4 years ago

How to run the different cloud providers? Should there be an example deployment for each?

I think we should stick to documenting one, AWS is probably the best example because of # of users that manage it themselves. The steps should mostly be the same across all providers as well

Do we need to account for environments where KCM and/or the CCM is/was running directly on a host as opposed to the k8s control plane?

I think we can assume control plane nodes are separate nodes

Are there dependencies on the cloud provider that need to be reconfigured? There is a cloudprovider.PVLabeler interface - how is this used? Are there some cloud providers that are also storage provisioners?

I think for the first pass, we should ignore storage providers and add CSI migration documentation iteratively.

Should monitoring topics be addressed?

No, I think just showing how to validate your CCM is working is fine

where should the documentation live

I think something like docs/tasks/administer-cluster/migrating-to-cloud-controller-manager is good.

onitake commented 4 years ago

I launched a PR, please submit corrections and input on how to migrate on AWS. I'm slightly biased towards private cloud CCM migrations, so please public cloud users: Give input on your cloud environment specifics.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

cheftako commented 4 years ago

/remove-lifecycle stale

cheftako commented 4 years ago

/lifecycle frozen

cheftako commented 4 years ago

/cc @jiahuif

cheftako commented 4 years ago

/assign @jiahuif

cheftako commented 3 years ago

/cc @jpbetz