search

kubealex / libvirt-k8s-provisioner

Automate your k8s installation
MIT License
425 stars 76 forks source link

Feature request: Support deploying multiple clusters to the same host #54

Closed 3rd-st-ninja closed 3 years ago

3rd-st-ninja commented 3 years ago

Requesting support to deploy multiple clusters to the same KVM host. My thoughts on how this could be accomplished:

  1. Terraform state files are generated in the files/terraform/* directories for each playbook. When deploying a second cluster, terraform wants to destroy the existing cluster. A way around this - assume a cluster name is test1. Ansible creates a dir clusterconfigs/test1 and copies terraform/files/* there then uses that path to deploy the test1 cluster. All the state files for each cluster would be in their own sub dir and would prevent conflicts when creating additional clusters.

  2. dnsmasq files placed into /etc/NetworkManager/dnsmasq.q/ and /etc/NetworkManager/conf.d/ would have to be cluster-specific and avoid conflicting if deploying multiple clusters in the same domain (e.g. named libvirt_clustername-domainname_dnsmasq.conf and localdns-clustername-domainname.conf respectively. Though I'm not 100% certain about the localdns-* configs). The contents of the dnsmasq config file would have to be something like server=/clustername.domainname/networkcidr to avoid conflicts with existing clusters in the same domain as well as preventing the kvm host from conflicting with the domain's authoritative name server if one exists.

3rd-st-ninja commented 3 years ago

The kubeconfigs that are copied from the master for each cluster will also have to be renamed per-cluster. I did some testing with this and SSH keys and came up with the following:

copy kubeconfigs to ~/.kube/<cluster-name>-<domain>.config symlink SSH keys from clusterconfigs/clustername/id_rsa* to ~/.ssh/id_rsa_<cluster-name>-<domain>*

The SSH key symlink was more of a convenience than anything. the cleanup script would have to be modified to handle all the file name changes I mentioned in this request as well. One "gotcha" I ran into was that the cleanup script removes terraform, which means if I wanted to remove a second cluster, it would fail. So I left terraform intact to avoid that situation.

kubealex commented 3 years ago

Thanks a lot for the input, I implemented it exactly as you suggested, just avoiding polluting user's home with keys/kubeconfig.