Is the hetzner cloud provider stable yet?

User Story

While working on my new blog article about kubermatic and hetzner I found a few issues, that created the feeling that Hetzner support in Kubermatic is not yet stable.

You can find a first draft of this article here: https://github.com/shibumi/shibumi.dev/blob/master/content/posts/kubermatic-on-hetzner.md

Here are the problems I have encountered so far:

Missing annotations in the nginx-ingress-controller service The kubermatic-installer does not automatically set a hetzner load balancer annotation to the nginx-ingress-controller. We might can fix this via modifying the nginx-ingress-controller values, but I don't know how to change this in the kubermatic-installer. If you can point me to a specific line of code from where I can start I am happy to provide a PR for additional ingress annotations :) In the end it should be possible to add custom annotations in the kubermatic.example.yaml file. I think this should be useful for other cloud providers as well.

Another Problem I encountered is cluster communication. I managed to create a cluster in the UI, but the cluster creation failed:

chris motoko ~/kubermatic 17:25:59 a9833cb1 master  kubernetes-admin@master cluster-xbrkdflsmq
❯ kubens cluster-xbrkdflsmq
Context "kubernetes-admin@master" modified.
Active namespace is "cluster-xbrkdflsmq".
chris motoko ~/kubermatic 17:26:04 a9833cb1 master  kubernetes-admin@master cluster-xbrkdflsmq
❯ kubectl get pods
NAME                                              READY   STATUS     RESTARTS   AGE
pod/apiserver-7b9456c7cd-tmv7l                    0/4     Init:0/1   0          2m56s
pod/apiserver-7b9456c7cd-wzl9z                    0/4     Init:0/1   0          2m56s
pod/controller-manager-658ff44b89-jm4bv           1/2     Running    1          2m56s
pod/dns-resolver-57fb778488-drklk                 2/2     Running    0          2m56s
pod/dns-resolver-57fb778488-lgl28                 2/2     Running    0          2m56s
pod/etcd-0                                        0/1     Pending    0          2m56s
pod/etcd-1                                        0/1     Pending    0          2m56s
pod/etcd-2                                        0/1     Pending    0          2m56s
pod/kubernetes-dashboard-7dd5b4799c-9h2j2         1/1     Running    0          2m56s
pod/kubernetes-dashboard-7dd5b4799c-wwxsm         1/1     Running    0          2m56s
pod/machine-controller-74fd44d686-zf7vl           1/1     Running    4          2m56s
pod/machine-controller-webhook-7cfb964d5b-l5vsl   0/1     Running    1          2m56s
pod/metrics-server-5bb8f467fb-g6qpp               3/3     Running    0          2m56s
pod/metrics-server-5bb8f467fb-vvjlh               3/3     Running    0          2m56s
pod/openvpn-server-f587f95c5-27xj8                3/3     Running    0          2m56s
pod/scheduler-55f9c9764c-zjnz2                    1/2     Running    1          2m56s
pod/usercluster-controller-84879dc867-jh8tb       0/1     Running    0          2m56s

With the following logs for the etcd-running init container in the apiserver pod:

{"level":"warn","ts":"2021-05-30T15:27:10.313Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-3c76155a-4268-42d4-b6da-e87113a510bb/etcd-0.etcd.cluster-xbrkdflsmq.svc.cluster.local.:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp: lookup etcd-1.etcd.cluster-xbrkdflsmq.svc.cluster.local. on 10.96.153.22:53: no such host\""}
Error: context deadline exceeded
waiting for etcd

Do you have any idea? This leads to my final question:

How stable and tested is the hetzner provider? Will I have a better experience in my company with the vsphere provider?

Acceptance criteria

kubermatic / kubermatic

Is the hetzner cloud provider stable yet? #7146