billimek
commented
3 years ago This was created for future reference, closing now!
Closed billimek closed 3 years ago
billimek
commented
3 years ago This was created for future reference, closing now!
Background
Due to the issues in #161, I had an unplanned opportunity to test rebuilding and restoring data for a new kubernetes cluster. The previous cluster has been running for over a year, and there was bound to be some unintentional drift from code persisted in github.
Part of the goal was to switch the k3s networking CNI from the default flannel to calico.
Calico experiment
Shut-down all k3s nodes & VMs
Using proxmox, created a new k3s master VM via these instructions
Looking at the calico documentation, it suggests that you only need to apply a couple of manifests to install calico. However, the Rancher documentation suggests that you need to _Modify the Calico YAML so that IP forwarding is allowed in the containersettings section. Unfortunately, it was not clear how or where to make this setting.
I tried to create a k3s master node with flannel disabled and used the 'Auto Deploying Manifests' feature by putting the two Calico manifest yamls into the
/var/lib/rancher/k3s/server/manifestsdirectory on the master node.This seemed to install something, but there were still issues. The provided
custom-resources.yamlassumes that the cluster CIDR block is192.168.0.0/16and would require to be changed from the default if using a different cluster CIDR. Additionally, it appears that the calico customallow_ip_forwardingsetting needs to be set somehow.Given that there appears to be increasing complexity getting calico installed via a cloud-init configuration, I opted to stick with the default flannel for now and tackle calico at a later date, assuming the cluster restore isn't too painful.
Rebuild Cluster
VMs
Using proxmox, created a new k3s master VM via these instructions
Grabbed the k3s token and generated some new k3s VMs and they auto-joined the new cluster. Ensured that the new k3s VMs used the same mac address as the old for ease of transition.
bootstrapping cluster
Took the opportunity to freshen-up the rook configuration as this will be a brand-new install of rook
Running bootstrap-cluster.sh revealed some issues already identified in #157. I don't do ansible and didn't want to install the prometheus CRDs out-of-bound (yet), so I disabled the flux and HelmOperator ServiceMonitor stuff
Vault had a circular reference to itself when I was experimenting with ingress. Vault should not rely on itself, obviously, and I wasn't doing the ingress, so removed the vault self-reference
Flux HelmRelease needed CRDs present before it would install, so I create the namespace and install the CRDs as part of the bootstrapping now
After this point, everything basically installed as expected. There were a couple of minor tweaks needed but nothing too problematic
Restoring workloads
With all of the workloads installed the following persistent data needed to be restored from velero backup:
The restore process, for each of these, was basically (using
unifias an example)kubectl delete deployment unifikubectl delete pvc unifivelero restore create --from-backup velero-daily-backup-20201114060055 --selector "app.kubernetes.io/name=unifi"The above took a large bulk of time because of the number of workloads to restore. It eventually all completed and required some babysitting
In the future I would like to automate the restoration process