terragrunt plan-all - Add option to apply dependencies

[markhopwoodnvm]

We're implementing a pipeline around our terraform builds where each module has a CI build which runs a plan and apply. We'd like to be able to use this model on modules that have dependencies on other modules. This would require all dependencies to be applied prior to running a plan on the module we are testing. Using the example from your documentation :

root ├── backend-app │ ├── main.tf │ └── terraform.tfvars ├── frontend-app │ ├── main.tf │ └── terraform.tfvars ├── mysql │ ├── main.tf │ └── terraform.tfvars ├── redis │ ├── main.tf │ └── terraform.tfvars └── vpc ├── main.tf └── terraform.tfvars

with the following dependencies between Terraform modules:

backend-app depends on mysql, redis, and vpc frontend-app depends on backend-app and vpc mysql depends on vpc redis depends on vpc vpc has no dependencies

terragrunt = { dependencies { paths = ["../vpc", "../mysql", "../redis"] } }

A terragrunt plan-all (with apply dependencies opiton) for backend-app would run the apply on vpc, mysql and redis, then a plan on backend-app

[brikis98]

I think the expectation for the plan command is that it's read-only and therefore always safe to run. It may be a bit surprising for it to be able to apply changes, even if you ask for them explicitly via a flag.

Perhaps the proper abstraction for this is an apply-all-dependencies command that runs apply-all on the dependencies of the current module, and their dependencies recursively, but not the current module?

[markhopwoodnvm]

Yep, that sounds good

[vas1468]

Can we give some love to this enhancement please? No one is assigned yet :( Unfortunately I cannot help.

[w32-blaster]

+1

[Hmerac]

I also encountered with this, some of our modules have dependencies over other modules.

The problem is, at first-run of terragrunt plan-all phase, dependent modules aren't able to retrieve output variables from other module's remote state file because terragrunt plan-all doesn't initialize state files as you know.

We needed to specify terraform apply for each module with regard to dependencies. Even this doesn't solve the fail problem when you run terragrunt plan-all at first-run.

So -> +1 🙌

[brikis98]

It says it right in the README:

Note: It is important to realize that you could get errors running plan-all if you have dependencies between your projects and some of those dependencies haven't been applied yet.

Ex: If module A depends on module B and module B hasn't been applied yet, then plan-all will show the plan for B, but exit with an error when trying to show the plan for A.

[brikis98]

Hm, that's weird. I was responding to a comment that's now gone... Uh... I guess the commenter deleted it?

[Xtigyro]

Hey folks - has there been any progress since the last comment?

[brikis98]

Not that I'm aware of. PRs still welcome.

[derekrprice]

Not sure how complicated this would be, but it seems like the right way to approach this would be not to actually apply the dependencies, but to fake it. As each level is planned, the desired state is already output. That desired state could be used to create a mock remote state used by each dependent plan. Terragrunt is already manipulating remote state in order to share a single definition block down the tree. Is there a way that it could substitute a "terragrunt-mock" state backend provider in the specific case of plan-all?

In my defense, if this is a crazy idea, I don't have any direct experience with the Terragrunt code base or Go yet. Maybe soon, if I keep finding issues like this. :)

[brikis98]

@derekrprice Don't think that would work. Figuring out the right fake outputs to return for every modules is hard enough, but if we're just returning mock data, then when those outputs are fed as inputs to other modules so those modules can run plan, they'll hit errors because those outputs won't correspond to real resources, so all the API calls will fail.

[derekrprice]

@brikis98 Vanilla Terraform can do it when all the resources are in the same state file. It seems like it should be possible. Not easy, perhaps, but possible.

[trallnag]

@derekrprice I feel like this would be out of scope for Terragrunt. It basically means reinventing Terraform

[dmead]

I could really use this feature.

I have a module that when applied pushes some resources to a kubernetes cluster.

it's dependency is responsible for grabbing an auth token which expires at regular intervals.

when that token expires, i can't plan on the module in question.

I really like the idea of plan the current module but apply the dependencies. otherwise i have to string together some bash to always apply the thing.

[marshall7m]

Here’s a Terraform module that detects all changes within Terragrunt dependency chains and orchestrates a continuous deployment flow that respects the dependeny chain order. The module offers AWS account-level dependencies and approval requirements. In addition, it includes handling for rolling back newly deployed Terraform provider resources. See the README.md for the full description of how this all works here.

Side note: I’m the owner of this repository. If you do check it out, I'm open to any questions and brutally honest feedback!

[ikegentz]

It's not a perfect solution (IMO a "perfect" solution is way beyond the scope of Terragrunt), but the mock_outputs feature of Terragrunt is about as close as you can get to solving this without re-inventing the wheel, but this still doesn't solve the entire issue. Sometimes there are modules that have a data resource in them which will always run and need actual outputs (not just mock outputs), so the use-case that OP is suggesting is still valid. Otherwise as @dmead mentioned, you have to write some bash-foo to achieve this (we're using makefiles)

[fraz78]

It's not a perfect solution (IMO a "perfect" solution is way beyond the scope of Terragrunt), but the mock_outputs feature of Terragrunt is about as close as you can get to solving this without re-inventing the wheel, but this still doesn't solve the entire issue. Sometimes there are modules that have a data resource in them which will always run and need actual outputs (not just mock outputs), so the use-case that OP is suggesting is still valid. Otherwise as @dmead mentioned, you have to write some bash-foo to achieve this (we're using makefiles)

I'm having this same issue with a data call when running a terragrunt run-all plan. Any advice?

[ikegentz]

@fraz78 Since the time I had made that comment we just accepted that run-all plan doesn't work if you haven't already done an initial run-all apply, and just used Terragrunt as such. That said, the only real suggestion I'd make is to create a makefile/script of some sort in each of your terragrunt directories that does an apply on any resources you have identified as causing the issue. Then you run make plan-all (which under the hood does those applies/plans as necessary.

Not trying to dissuade you from attempting that implementation, but we found the juice wasn't worth the squeeze, and just didn't try to use terragrunt that way.