Exposing microK8s to cloud load balancers

[worldofgeese]

MicroK8s is running on 3 EC2 instances in HA mode with Traefik 2 as ingress controller stuck with <pending> for it's External IP.

I spent all of Friday writing Terraform rules to create an AWS Network Load Balancer and now I'm uncertain if microK8s supports even the minimum --cloud provider: external flag in which to interface with these externally provided resources. I'm curious if others have gotten this to work. To be clear, I'm using Traefik 2 installed with helm3 but you could test this using the native ingress addon too. You can add this flag to /var/snap/microk8s/current/arg under the kubelet, apiserver and controller files then restart microK8s with microk8s stop and microk8s start.

Now I'm wondering first of all if adding an AWS NLB using Terraform or manually as demonstrated here is even supported and, if it is, if there is perhaps a simpler way, in terms of reducing my own maintenance overhead, such as installing Juju and using its aws-integrator charm, which includes support for AWS Network Load Balancers (provisioning too?). This tutorial uses the kubernetes-core bundle in addition to microK8s to enable AWS integration. It's unclear if it works with exclusively microk8s clusters as in my scenario.

To be clear, --cloud-provider=aws would be the simplest solution but is unsupported by microK8s and will result in your node failing to switch to Ready status.

One question users might ask is, "why don't you just use EKS?" To which my answer is, I'm exploring a full-stack Canonical product pipeline (with an eye toward purchasing Ubuntu Advantage) using minimum viable resources on Big Cloud.

[worldofgeese]

Do you have any experience with Juju charms interacting with microK8s @ktsakalozos?

[ktsakalozos]

Hi @worldofgeese

MicroK8s is a certified Kubernetes distribution and does not drop any features, so everything that is supported upstream is also available in MicroK8s.

This tutorial uses the kubernetes-core bundle in addition to microK8s to enable AWS integration.

This is not correct. The tutorial you point at does not use MicroK8s. It uses Charmed kubernetes. Charmed Kubernetes is the enterprise grade distribution by Canonical. The aws integration charm you are referring to has to be used with Charmed kubernetes and not MicroK8s, see the official docs.

MicroK8s does not offer any easy to use integrators so you will need to follow manual steps. If you point me to what official upstream docs you are following I might be able to reproduce the issues you see.

[worldofgeese]

MicroK8s is a certified Kubernetes distribution and does not drop any features, so everything that is supported upstream is also available in MicroK8s.

I believe you but then we need to square away why MicroK8s doesn't automatically provision a cloud provider's load balancer for LoadBalancer types. The official Kubernetes docs indicate CNCF compliant distros should automatically provision on supported cloud provider backends, of which AWS is one. I admit I could be conflating the existence of upstream's Cloud Controller Manager with compliance.

Any LoadBalancer type I spin up on my cluster is never given an IP: it is eternally <pending>, indicating MicroK8s does not auto-provision an AWS load balancer. Indeed, nothing suggests a provisioned cloud load balancer in my AWS console.

MicroK8s does not offer any easy to use integrators so you will need to follow manual steps. If you point me to what official upstream docs you are following I might be able to reproduce the issues you see.

An official tutorial for adding a MicroK8s cluster to a Juju "cloud" and then deploying charms to that same cloud is here. It's reasonable to assume other Juju charms, like the aws-integrator charm, should "just work" with a MicroK8s cluster.

[ktsakalozos]

There has to be a service/daemon that would read the requests for LB IPs and will interface with the respective cloud to satisfy these requests. Maybe you need to look at https://github.com/kubernetes-sigs/aws-load-balancer-controller. @johnsca may know more on the exact configuration you need to follow.

I see your point on the docs. @evilnick may be able to forward this comment to the right channels.

[joshbranham]

We are using the alb-ingress-controller aka pre-2.0 aws-load-balancer-controller as well as kind: Service, type: LoadBalancer in EC2 by setting the --cloud-provider=aws in the various k8s systemd unit args files

[worldofgeese]

We are using the alb-ingress-controller aka pre-2.0 aws-load-balancer-controller as well as kind: Service, type: LoadBalancer in EC2 by setting the --cloud-provider=aws in the various k8s systemd unit args files

Did you only need to add --cloud-provider=aws to /var/snap/microk8s/current/arg/{kubelet,kube-apiserver,kube-controller-manager}?

[joshbranham]

I had to do more than that, the provider ID was needed too (which I get from ec2 metadata)

microk8s kubectl patch node \
    "$host" \
    --patch '{"spec":{"providerID":"'"$providerid"'"}}'

echo "> Setting command line args"
echo "--cloud-provider=aws" >> /var/snap/microk8s/current/args/kube-apiserver
echo "--cloud-provider=aws" >> /var/snap/microk8s/current/args/kubelet
echo "--cloud-provider=aws" >> /var/snap/microk8s/current/args/kube-controller-manager
echo "--provider-id=$providerid" >> /var/snap/microk8s/current/args/kubelet

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bessonov]

activity

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ysaakpr]

I think this need to be re opened. MicoK8s is a fantastic way to operating a mini k8s cluster for various use cases. And one of the common usage is to deploy them on a single node or HA nodes in cloud computes like AWS EC2. It would be a good to have a support for connecting cloud load balancer to a service.