Direct terrraform-waypoint integration

[yuriy-yarosh]

Is your feature request related to a problem? Please describe.

I'd like to be able to run a separate development infrastructure setup instead of what Waypoint already doing for ECS and Nomad - configure any number of 3rd party tools with Terraform and pass the respective terraform managed resources from the terraform state to the ECS / Nomad plugins. Ideally, make those plugins integrated with the existing ECS / Nomad terraform providers.

This should've prevented a ton of runner / project lifecycle bugs that we can observe right now.

Ideally, the waypoint installation should be managed by the waypoint itself, with all the update lifecycle and namespaced project hierarchy that is projected onto the existing organizational structure (i.e. AWS Organizations, AFT and Control Tower etc), so every dev could get an isolated and downscalable development environment, maybe with a bunch of shared resources (AWS RAM bindings etc).

There's also a bunch of chiken-egg types of problems when managing various setups on Kubernetes, like setting up Vault with Helm, then configuring it with terraform, and continue to deploy Vault-dependent services with Helm or other tool (manual multi-stage deployments). Using terraform apply -target is too tedious... and it's a well known Terraform design flaw (more on that later).

Describe the solution you'd like All in all there's a clear distinction between Infra GitOps using Terraform IaC, and development GitOps using Waypoint or ArgoCD. But we're really missing a proper way to trigger infra GitOps or amend some of it's settings (terraform variables) for a complete feedback loop. That's why projects like crossplane had emerged as a terraform wrapper, in the first place.

It would be great if waypoint could do something similar with a dedicated Terraform Plugin, and solved this issue.

Describe alternatives you've considered There's none, because it involves waypoint design flaws or just misplaced priorities ...

Additional context I get that no one is willing to admit that Waypoint is just a replacement for ArgoCD. with a half-backed HashiCorp stack integration, at this point, and a source for upsales, but I see really frustrated people around, and I get that there are numerous ways to monetize that frustration.

I mean, like I get that HashiCorp has business priorities and fixing design flaws, that really detracts their own Clients and slows down Community Adoption greatly, is not one of them, for sure.

For instance, managing everything with terraform is hard because HashiCorp had neglected it's Deferred Provider Initialization Design Flaw since 2015, so people were forced to adapt and -target everything manually. I can rant about it all day long... and had been looking for a workaround for a very long time, hopefully Waypoint might be a good pick for that. At least if there were a dedicated Terraform plugin - waypoint could do all the relatad -target'ing.

One provider can't depend_on another and that "future version" never came to be.

I do expect this issue to be neglected as well.

[voycey]

Hi Yuriy - I would like to add that this issue: https://github.com/hashicorp/waypoint/issues/4148 and its associated ones relate to this, currently we are trying to hack this together but limitations in Waypoint prevent us from being able to do this. We have currently settled on using Vault as a source of truth for the various items created by Terraform to create a config that can be used by items deployed by Waypoint (albeit not using Waypoint - which is frustrating).

[yuriy-yarosh]

@voycey I'm working on a custom terraform controller, somewhat similar to tf-controller and Crossplane, - the general idea is to standardize a yaml config for the whole infra and organization structure, update it on the fly from a set of self-managed CRD's (one CRD per TF Module).

So, it could work the same way as terraform-consul-sync, but on Kubernetes, driven with a custom autoscaler like Keda. That will complete the feedback loop between observability and infra scaling / capacity planning for kubernetes and Terraform. I.e. you won't need Karpenter, per se, and could get the same features for any other hosting provider with minimal effort.

I've decided to skip on Waypoint completely, due to it's limitations and manage only infra with Terraform, including all the kuberentes manifests... ArgoCD can manage all the CRDs for my custom Terraform Controller, and that's just perfect.

I'm also planning to contribute depends_on for_each support for Terraform providers, although it could change the Plan format, there should be some form of migration mechanism for backwards compat (didn't dig into it, yet).

[catsby]

Hello @yuriy-yarosh - thank you for taking the time to write this issue out and share your thoughts on where Waypoint is today, and where Waypoint and Terraform integration improvements are needed. Having personally worked as an engineer on both Terraform and Waypoint here at HashiCorp, I understand some of the frustrations and possible confusion regarding the overlap of functionality between the two products.

Internally we’ve been discussing ideas about where and how Waypoint should integrate with Terraform. Early in the project, we wanted Waypoint to be able to stand alone without a dependency on Terraform (explicit or implicit). Some people may discover Waypoint with no knowledge of any other HashiCorp product, and we want those users to have a good experience without having to learn multiple tools. Other people may be experienced Terraform users and are likely confused or frustrated as to why Waypoint seems to be doing things its own way and not benefiting from Terraform. We agree that there is a pain currently with Waypoint not having a tighter and more useful integration with Terraform! Currently, our goals this year as a team are to provide a better value with Waypoint by leveraging Terraform where it makes sense.

In the future we hope to make the integration points between Waypoint and Terraform more clear or obvious. As a specific example, we recently improved our Terraform Cloud config sourcer to be able to read multiple Terraform Cloud outputs in a single block, and reference those in a waypoint.hcl file. The idea here being to simplify the integration such that a base level of infrastructure could be created in Terraform Cloud, and consumed by Waypoint and used as inputs. The ideal end result would be Waypoint itself will only modify the infrastructure where absolutely necessary for a given deployment, for example launching new ECS tasks or modifying target groups. This functionality exists today but it’s a bit tedious and possibly error prone for the users.

I realize this reply does not explicitly address all your specific items here or mentioned in other comments, but I hope to assure you that we are grateful for the feedback we receive, even if we don't immediately respond or act upon it. A better story around Waypoint and Terraform is in the works, we’re just not at a point we can share specifics yet. I see in your latest comment too that you’ve decided to skip on Waypoint completely, and it sounds like you have a solid plan for moving forward the way you want to. Hopefully someday Waypoint will be something you’ll consider again, but until then we wish you luck and again thank you for your feedback.

[voycey]

@catsby - the main USP of Hashicorp is that HCL drives everything you do here. However in Waypoint its simply not the case, its often HCL with conditions, the string variables thing you mentioned is one of these conditions and it is great that it is being fixed, but also being able to dynamically source config is a massive feature of HCL that Waypoint doesn't support (https://github.com/hashicorp/waypoint/issues/4148) - hopefully this can be focused on next as that way there doesnt need to be a direct integration with Terraform as it means we can create that integration ourselves.

Very much onboard with Waypoint being standalone - but we pick the Hashicorp ecosystem for a reason and Waypoint seems to forget this