Test CCM/CSI migration -- Test Release 1.4

[xmudrii]

Instructions:

• Download the latest KubeOne 1.4.0 release candidate
• Follow the Create a Kubernetes cluster tutorial to create your cluster
  • Make sure to have .cloudProvider.external disabled:

    cloudProvider:
    external: false

• Wait for machine-controller-managed nodes to join the cluster
• Ensure all pods are Running
• Create a LoadBalancer Service and point to some test pod (e.g. Nginx pod). The LB should be reachable and serve the content as expected
• Create a GCE PV and attach it to the pod
• Follow the CCM/CSI migration tutorial to migrate your cluster
• Make sure to rotate your MachineDeployments afterward
• Wait for rotated nodes to join the cluster
• Ensure that the LoadBalancer service is still reachable
• Recreate the LoadBalancer service and ensure it's reachable
• Recreate the pod with the volume attached and make sure the new pod is Running

This test can be done on a single operating system with a single Kubernetes version, but each supported provider should be tested:

• [x] Azure
• [x] OpenStack
• [x] vSphere

It's recommended to run tests on Ubuntu with Kubernetes 1.23.3.

[SimonTheLeg]

@xmudrii What should I test on Azure? Reading through the migration guide it says

The CCM/CSI migration process is currently supported only for OpenStack and vSphere

[xmudrii]

@SimonTheLeg The document is outdated. The CCM/CSI migration is supported on Azure as well.

[SimonTheLeg]

okay worked without any issues on Azure 👍

@xmudrii should I update the guide or is there some centralized overhaul planned?

[xmudrii]

should I update the guide or is there some centralized overhaul planned?

I already did that in https://github.com/kubermatic/docs/pull/964.

[SimonTheLeg]

I have concluded the test on OpenStack, but found some things:

1. The old storageclass is not being removed and is also still set up as the default storageclass. This results in the following error, when you create a new pvc.

     Error from server (Forbidden): error when creating "pvc-azure.yaml": persistentvolumeclaims "nginx" is forbidden: Internal error occurred: 2 default StorageClasses were found

   Removing the defaulting annotation solves this issue. However I still think, kubeone should either clean up the old one automatically, or we should at least have a remark in the migration guide.

2. In the installer it says WARN[16:05:11 CET] If you currently use Neutron LoadBalancers, migrating to the external CCM/CSI will cause *ALL* Load Balancers to be recreated!. This does not seem to be the case. The LoadBalancer I created before the migration is still a Neutron LoadBalancer afterwards

   

3. Creating a new LoadBalancer fails after the upgrade with the following error

     Warning  SyncLoadBalancerFailed  111s (x7 over 7m12s)  service-controller  Error syncing load balancer: failed to ensure load balancer: error creating loadbalancer kube_service_kubeone-test-ccm-migration_default_nginx-new: error creating loadbalancer {"name":"kube_service_kubeone-test-ccm-migration_default_nginx-new","description":"Kubernetes external service default/nginx-new from cluster kubeone-test-ccm-migration","vip_subnet_id":"94f8a3df-355a-463c-8a49-74d3e1d82d76","provider":"amphora","listeners":[{"protocol":"TCP","protocol_port":80,"default_pool":{"lb_algorithm":"ROUND_ROBIN","protocol":"TCP","name":"TCP_80_pool","members":[{"address":"192.168.1.4","protocol_port":31142,"name":"kubeone-test-ccm-migration-pool1-6f9dc76fc8-k4xpq","subnet_id":"94f8a3df-355a-463c-8a49-74d3e1d82d76"}]},"connection_limit":-1,"timeout_client_data":50000,"timeout_member_data":50000,"timeout_member_connect":5000,"timeout_tcp_inspect":0,"allowed_cidrs":["0.0.0.0/0"],"tags":["kube_service_kubeone-test-ccm-migration_default_nginx-new"]}],"tags":["kube_service_kubeone-test-ccm-migration_default_nginx-new"]}: Bad request with: [POST https://api.dbl.cloud.syseleven.net:9876/v2.0/lbaas/loadbalancers], error message: {"faultcode": "Client", "faultstring": "Provider 'amphora' is not enabled.", "debuginfo": null}

[xmudrii]

1. I'm currently investigating this, but the problem appears to be more serious

2. 1852 extends this warning to mention that the Neutron Load Balancers must be removed manually

3. This issue is coming from OpenStack, so there's nothing we can do about it. It also happens on non-migrated clusters that use the external CCM

[xmudrii]

The first issue is fixed by #1853. I'm closing this issue as the CCM/CSI migration works as expected.