search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
8.09k stars 981 forks source link

terragrunt run-all graph plan/apply #3411

Open Yethal opened 2 months ago

Yethal commented 2 months ago

Summary

It would be useful to have ability to plan multiple modules and both their dependencies and dependants. Currently it's possible to plan multiple modules and their dependencies via run-all plan/apply but planning a module and its dependants via graph plan/apply is only possible on per module basis

Motivation

At my company on every PR we detect which modules have changed and then run plan on those modules only via run-all plan with multiple --terragrunt-include-dir arguments added. This works fine for planning changes in modules and their dependencies but tells us nothing about changes in downstream modules. While it would be possible to instead run graph plan in each of those modules individually such approach is inefficient. Let's assume a scenario in which a PR changes modules A and B and B is a dependency of A and has its own downstream dependencies. If we run separate graph plan commands in parallel then graph for module A will include module B as well and its dependencies and plan for module B will include module B and it's dependencies once again. As a result such job cannot be parallelized due to state locking. Running the plans in parallel but with max parallel set to one (to avoid issues with state locking) will cause the job to be very slow because the same modules will be planned multiple times. Alternatively we could run plan for all modules in the repository instead of only the changed ones via regular run-all plan however in repositories with hundreds of small modules this would take ages.

Proposal

Introduce new command run-all graph plan/apply This command would work the same as existing run-all commands but when used with terragrunt-include-dir it would not only add specified modules and their dependencies to the graph but also their dependants thus ensuring the plan shows full scope of the changes.

Technical Details

Basically, take the functionality already present in graph plan command and make it work with run-all --terragrunt-include-dir

Press Release

Terragrunt is now capable of planning and applying all nodes affected by particular change request.

Drawbacks

No response

Alternatives

Gather list of changed modules, for each modules run graph plan, wait a few hours and deduplicate the results

Migration Strategy

No response

Unresolved Questions

No response

References

No response

Proof of Concept Pull Request

No response

Support Level

Customer Name

No response

yhakbar commented 1 month ago

Please take a look at the RFC #3445. In that RFC, I'm proposing a redesign of the CLI that will make it so that run-all and graph will be replaced with --all and --graph flags on a new run command.

Initially, the objective would be to have them be mutually exclusive so that existing functionality is matched, but in the longer term, the behavior you're asking for might be addressed by that.

What do you think of that? If you're in favor of that, please share your feedback on the CLI redesign RFC, then consider updating this RFC to reflect that change in design.

bakavets-ma commented 3 weeks ago

+1 (terragrunt run-all graph plan/apply)