evaluate and build modern k8s infrastructure

nikhiljha commented 4 years ago

Things I don't like about the current way the OCF does deployment:

Using ruby templates makes it very difficult to test deployments without manually editing the files and filling in variables.
There is no good way to do Helm deployments.
The structure of the deployment is assumed (one namespace, prefixed with app-).
A lot of it is in-house tooling, which doesn't translate well to other k8s clusters.
- it's hard to get help with this stuff online
- it's potentially easier to upgrade if we're not maintaining everything
Secret management involves interesting tricks.
Making a test cluster or test namespace is a manual process.

What to do instead:

ArgoCD as a replacement for kubernetes-deploy/krane
- understands raw yaml files (we can keep most of our existing config)
- understands helm, kustomize, and all the other deployment methods (makes using all the existing helm charts potentially easier)
Jenkins is fine IMO, but a lot of people are using Drone for k8s -> we could evaluate it
Use a managed secrets provider instead of magic puppet directories.

Tasks (in this order more or less) (in an ideal world):

[x] Deploy ArgoCD directly to the cluster. done
[x] ... bonus points for High Availability (across 3 nodes, except this is fake since they're all secretly riptide)
[x] Configure keycloak login.
[ ] make sure everyone understands security implications
[x] Evaluate secrets providers, pick the best one OR figure out how to integrate our current system if they're all bad.
[x] Deploy a secrets provider.
[ ] Evaluate & deploy a CI solution, or keep using Jenkins
[x] Store the ArgoCD configuration in git (ArgoCD manages ArgoCD).
[x] tell github to send webhooks to argocd
[ ] Disaster recovery bootstrapping solution
[ ] Configure ArgoCD to use our existing prometheus deployment for metrics -> Grafana
[ ] Setup notifications
[x] write documentation for how to deploy
[ ] slowly start moving services over to an ArgoCD based deployment
[ ] delete all the legacy deployment tools, secrets management, CI/CD, etc

BernardZhao commented 4 years ago

As @Baisang mentioned, also documentation. I can help with that as well

nikhiljha commented 4 years ago

@fydai suggests using pulumi (actually he's already deployed it) instead of ArgoCD

pros:

it's just python, so we can just import parts of other deployments
- as fydai says: "no copy pasting YAML"
- instead we import antigravity
it can manage more than just k8s

cons:

no multi-user auth (we need to share a username and password or email pulumi and ask if they want to sponsor us)
no pretty web interface or control panel

nikhiljha commented 3 years ago

ocf / projects