Adam-D-Lewis
commented
1 year ago Looking into this further, ArgoCD would help with the issues described above, but it would be a pretty big process to migrate. Right now, Nebari stages are terraform modules. To use ArgoCD fully, we'd need to convert all the stages to helm charts. The helm charts would need to contain a terraform operator (e.g. https://github.com/GalleyBytes/terraform-operator or https://github.com/weaveworks/tf-controller) to deploy the cloud resources. We could still deploy in stages using ArgoCD Sync Waves. Because the effort needed to make all these changes is high relative to the benefits, I'm going to drop this for now.
Replace Deployment Action with ArgoCD
Summary
For CICD, Nebari supports Github Actions. To support other git platforms (e.g. Gitlab Runners, Azure DevOps, BitBucket Pipelines etc.) we have to port the github action over to the format accepted by those other platforms. Additionally, for private clusters where the nodes themselves are not publicly accessible, Nebari does not have a CICD solution since Github Action runners would not be able to publicly access the k8s cluster directly.
Additionally, if an admin wants to view the state of Nebari, they have to have k8s credentials which requires some manual steps. With ArgoCD, we could solve all the above problems. In this proposal, we deploy ArgoCD alongside the other components of Nebari via
nebari deploy. Then when changes are made to the deployment repo, ArgoCD will discover the changes and update the cluster configuration. Because ArgoCD uses a pull based approach rather than a push based approach, a private cluster will still be able to be managed via a GitOps approach.ArgoCD also provides a dashboard of the running resources where users could view pod logs and we could even give admins/developers the ability to modify k8s spec on the fly as admins and developers sometimes do during debugging in k9s currently eliminating the manual steps required by k9s mentioned earlier We can manage these permissions in Keycloak.
Other Benefits
Using ArgoCD also allows us to scope permissions more precisely. Currently, Github Actions has permission to modify the entire K8s cluster. With ArgoCD, we can scope permissions so ArgoCD can modify only specific K8s namespaces. ArgoCD is commonly used to manage environment promotion (dev -> UAT -> prod) (reference1, reference2) and this could possibly be our solution to doing this in Nebari as well.
User benefit
Design Proposal
We could follow the format of this repo and described in a recent CNCF talk though there are many articles about ways to integrate terraform and ArgoCD.
Alternatives or approaches considered (if any)
Best practices
User impact
Unresolved questions