add support for proxmox lxc containers

[acdoussan]

Proposed Changes

• adds support for using lxc containers as your kubernetes hosts
• https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185

Checklist

• [x] Tested locally
• [x] Ran site.yml playbook
• [x] Ran reset.yml playbook
• [x] Did not add any unnecessary changes
• [x] Ran pre-commit install at least once before committing
• [x] 🚀

[acdoussan]

Also worth noting: everything about this appears to be working, but I have not been able to reach a VIP assigned by metallb. Assuming this is a network configuration problem on my end though, since metallb does assign an IP as far as I can tell.

curl just waits forever, browser gives a timeout after spinning for a long time with http://10.0.3.1.

$ kubectl describe service nginx
Name:                     nginx
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         PreferDualStack
IP Families:              IPv4
IP:                       10.43.5.3
IPs:                      10.43.5.3
LoadBalancer Ingress:     10.0.3.1
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30932/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

$ kubectl get endpoints nginx
NAME    ENDPOINTS   AGE
nginx   <none>      92m

$ kubectl get services
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.43.0.1    <none>        443/TCP        100m
nginx        LoadBalancer   10.43.5.3    10.0.3.1      80:30932/TCP   96m

[timothystewart6]

Nice! Wouldn't this playbook work without modifications if an LXC container has an IP?

Also, would love to figure out how to test this with molecule and GitHub actions before merging if it's possible.

[acdoussan]

Nice! Wouldn't this playbook work without modifications if an LXC container has an IP?

In short, no. Without the updates to the conf file / adding the rc.local script, k3s crashes on startup.

Also, would love to figure out how to test this with molecule and GitHub actions before merging if it's possible.

Yeah, was going to look at that but I ran out of tinkering time and wanted to put something up to make sure you were open to it. I might have some more time this week, def this weekend, want to figure out why I can't reach the service first to make sure it is not something related to k3s setup. Will follow up by Sunday.

[acdoussan]

Alright, turns out not being able to reach the service was just another instance of me only half following the docs, and assuming the service file would also deploy containers. I now understand that is not the case, and you do in fact need to do both the service and the deployment 🙂.

With that being said, this does work and shouldn't need any additional implementation changes. Will still follow up by Sunday.

[acdoussan]

After doing a bit of research, I'm not totally sure if there is an easy way to test this with molecule.

There is a proxmox driver, but looks like that expects to be able to have access to a host already running proxmox, don't think that will fit in with the CI. Also looks like it only supports VMs, not containers.

I don't think there is much that could be done here, other than superfluous tests that check the file contents, similar to this. I don't think these add any value, and instead just add to the maintenance overhead. Ideally, we'd spin up a cluster with proxmox lxc containers and check that Kubernetes is running after deploying, but I don't see a simple path to make that happen currently. Thoughts?

Also, I spent some time playing with the permissions. Everything configured is definitely required, not running any of these tasks on either the control or worker nodes results in a cluster that does not start.

[timothystewart6]

Thank you for doing this. It's unfortunate that we don't have a test proxmox environment :)

[acdoussan]

Glad I could give back, hope it saves others some time :) thanks for making the slick playbook!