automate setup/teardown for testing cluster

[slugalisk]

To automate e2e testing we want to be able to spin up a multi-region kubernetes cluster in digitalocean. DO bandwidth pricing makes it a nonstarter for production deployment so while they do offer a hosted k8s solution it may make sense to roll our own especially if this comes with significant vendor lockin. We should investigate and decide whether the tradeoff for reusability vs ease of use makes sense...

• [x] investigate do hosted solution

cluster services:

• [x] prometheus
• [x] grafana
• [ ] cmd/svc

[x-Xymos]

I dunno about digital ocean but spinning up your own kubernetes cluster on a node/VM takes seconds, then all you gotta do is spin up some slave nodes and join the cluster. I imagine a self managed solution will be a lot cheaper

[slugalisk]

@x-Xymos perfect, can we work on this next?

[jbpratt]

https://github.com/containerd/containerd/issues/3291 Blocking pulling directly from GitHub's registry, we are trying to work around this currently pushed to a private DockerHub repo for now.