terraform and terraform-consul-sync support

[yuriy-yarosh]

Summary

Add an ArgoCD sidecar plugin for either terraform or terraform-consul-sync (CTS) support.

Motivation

All my yamls are in tf already, I'd like yet another config management integration, with the respective infrastructure state atomicity and locking. tf-controller is somewhat still raw, and I'm not a huge fan of fluxcd subsystem either.

Consul Terraform Sync (CTS) allows to replace simplest operators based on keda observability i.e. keda can modify the consul k/v and trigger the respective terraform infra change via cts. It's much more flexible and simpler than Kudo. Well, kudo needs some refactoring as well, so keeping the tool count minimal is always a good option.

Pretty common operators, especially database ones, could be replaced with a terraform module this way.

Proposal

Add a sidecar plugin with either terraform or terraform-consul-sync for terraform config. All the kubernetes provider managed resources could be exported directly from the terraform state, alongside the resulted resource definitions.

[yuriy-yarosh]

There could be also a docker provider integration there, as well.

[RossComputerGuy]

I've been looking into this since I'm looking to deploy AWS and Kubernetes things through ArgoCD. I've determined that terraform plan --out=[file] can output the stack in a weird binary format then use terraform show -json [file] to get a JSON formatted version of that. A sidecar would have to build YAML files unless there's a way to override the apply.

[yuriy-yarosh]

@RossComputerGuy there are couple of limitations and caveats - the exact pending terraform plan shouldn't break the argocd deployment itself.

My primary motivation is to simplify operator/CRD deployments by replacing them with Terraform Sync. Most of the common Grafana / Prometheus operators are far from anything I'd call stable, per se.

A sidecar would have to build YAML files unless there's a way to override the apply.

TF Kubernetes provider itself gives you yaml.

[adamdecaf]

We plan/apply using the native Kubernetes TF provider so being able to use that without a rewrite would really help us. Terraform also serves as a fallback if our ArgoCD deploy has issues or during disaster recovery planning - as we need to simulate partial failures within our environment.

[yuriy-yarosh]

@adamdecaf helm provider is somewhat in semi-abandoned state due to current waypoint marketing. I'm facing some minor bugs in Kubernetes TF provider from time to time, but the primary issue is that there's no direct k8s API codegen which is preventing adoption of 1.24+ API's, there are simply gaps and improper schemas in the current implementation.

For now I'm considering a couple of approaches for unified GitOps pipeline, where there's no strict division between Application and Infra GitOps.

1. Provision infra with TF
2. Render some custom Dockerfile Templates with Terraform to get rid of repeating Dockerfile chunks
3. Build images using Waypoint (docker tf provider needs a complete overhaul)
4. Deploy Kustomize or Helm charts with Waypoint
5. Run some tests using Waypoint

So it's Terraform -> Waypoint combo, which is somewhat competing with common Argo/Flux setups. I'd like to rewrite the existing docker provider into something Kaniko / buildkit like, or add some plugable builders... but not sure if it's a viable longterm option due to tight TF lock-in and a CD dependency.

Also I'm considering replacing CD part with some dagger.io or make a dagger tf provider... it also make sense to implement img builds and custom templating there too. I'll think about dagger<-> tf interop.

From design perspective ArgoCD <-> dagger <-> TF would be ideal for multistage GitOps deployments, where mixing multiple IaC tools like Crossplane + Terraform is not a viable option.

Building a custom img builder plugin for dagger makes more sense then rewritting the existing tf providers (less codegen, more substance). On the other hand, getting rid of templaty K8S TF provider codebase with some codegen would be really nice otherwise (there's not much substance i.e. custom complex logic there, just a bunch of typed API's and timeouty circuit-breakery thingies).

Thoughts appreciated.

[adamdecaf]

Interesting.. I'm not sure why you need an additional tool between ArgoCD and TF. It seems that's where both tools should work together better.

[blakepettersson]

It seems that's where both tools should work together better.

I agree that there should be more integration between the two tools, but that should happen on the Terraform (or maybe opentofu) side IMO.

Will close this for now since the prospect of this being worked on is quite low.