(feat): Monorepo build options idea

[lukasmrtvy]

Hello, Would be great to have something for handling monorepo use cases ( especially in CI ):

Folder structure:

azure(dir)
   westeurope
      project1
          main.tf
          terragrunt.hcl
      project2
          main.tf
          terragrunt.hcl
      project3
          main.tf
          terragrunt.hcl
      terragrunt.hcl
aws(dir)
gpc(dir)
terragrunt-definition.file(*)
terragrunt.hcl

Use cases:

1. User wants to trigger terragrunt apply project1 from definition file
2. User wants to trigger terragrunt apply all-azure-westeurope from definition file
3. User wants to trigger terragrunt apply multicloud-specific-all-projects from definition file
4. CI pipeline wants to determine which projects were impacted from definition file

terragrunt-definition.file(*) ( this will handle points: 1-3 )

terragrunt {}
projects {
  - name: project1
     path: azure/westeurope/project1
  - name: project2
     path: azure/westeurope/project2
  - name: all-azure-westeurope
     path: azure/westeurope
  - name: all-multicloud
     path: .
}

there should be also command like: ( this will handle point: 4 ) terragrunt get-diff which should output impacted projects, then I can automatically determine imapcted projects ( in PRs ) and run validate/plan only on these projects.

Of course this should be possible to handle with some sophisticated monorepo build tool like: https://github.com/korfuri/awesome-monorepo#build-systems--dependency-management-tools , but point is to have similar feature in terragrunt. ( not sure if is possible to handle it in non git based environments, maybe calculating sha, inotify, etc.. )

AFAIK https://www.runatlantis.io/docs/custom-workflows.html#use-cases can handle this monorepo problem.

Any ideas?

Thanks

[brikis98]

Use cases:

1. User wants to trigger terragrunt apply project1 from definition file
2. User wants to trigger terragrunt apply all-azure-westeurope from definition file
3. User wants to trigger terragrunt apply multicloud-specific-all-projects from definition file
4. CI pipeline wants to determine which projects were impacted from definition file

I think I'm missing some context. Why do you need a definition file here, especially for the first 3 use cases? Why can the user not run apply or apply-all in the corresponding folders directly? And how does a definition file help determine what projects were impacted (by commits, I assume you mean?).

[dudicoco]

@lukasmrtvy the way we do it is with git diff to find the directories contents that were changed in the commit, and then run terragrunt apply-all --terragrunt-strict-include --terragrunt-include-dir "path/to/dir1" --terragrunt-include-dir "path/to/dir2".

[nabladev]

Use cases:

1. User wants to trigger terragrunt apply project1 from definition file
2. User wants to trigger terragrunt apply all-azure-westeurope from definition file
3. User wants to trigger terragrunt apply multicloud-specific-all-projects from definition file
4. CI pipeline wants to determine which projects were impacted from definition file

I think I'm missing some context. Why do you need a definition file here, especially for the first 3 use cases? Why can the user not run apply or apply-all in the corresponding folders directly? And how does a definition file help determine what projects were impacted (by commits, I assume you mean?).

@brikis98 Because I don't want the user to do that, I want Jenkins to do that for me. Consider a scenario where everything is on the same repo (both modules and the terragrunt configuration for deploying several different envs into several different regions):

1. ops engineer makes a pull request to say master
2. jenkins triggers: plan -> comment pr with plan output
3. pr gets approved and merged
4. on merge jenkins runs apply

On point 2,4 how can Jenkins know what modules to plan/apply? Why should Jenkins know that? Because https://github.com/gruntwork-io/terragrunt/issues/720 yorinasub17 explains the drawbacks of *-all commands.

The only way for Jenkins to know that would be to dig through the pr and find out which modules have been modified and consequently run terragrunt plan/apply only inside the right folders.

[brikis98]

The only way for Jenkins to know that would be to dig through the pr and find out which modules have been modified and consequently run terragrunt plan/apply only inside the right folders.

Yup, that's more or less what we do. Alternatively, you can allow devs to specify what to deploy in the commit message, and have Jenkins parse that.

[arivictor]

@lukasmrtvy the way we do it is with git diff to find the directories contents that were changed in the commit, and then run terragrunt apply-all --terragrunt-strict-include --terragrunt-include-dir "path/to/dir1" --terragrunt-include-dir "path/to/dir2".

@dudicoco are you doing this manually or automated? I'm not sure how to handle when a module has been removed with the intent of it being destroyed. It won't exist in the PR or after merge so how do you go about handling this sort of logic through automation?

The only way I can think is to check the git diff main --name-status (or similar) on the pull-request branch, and if the file is deleted I'd then need to switch back to the main branch and run plan --destroy (for PRs).

Preferably running run-all would be preferable but the way this team has it setup its not possible.

[dudicoco]

@arivictor this is fully automated for creation/modification, however, we are currently destroying manually as it's a very sensitive operation.

If you would like to automate destroying I suggest the following solution:

1. Add the prevent_destroy = true flag in each terragrunt.hcl file
2. In the CI, detect when the flag is changed to prevent_destroy = false and run terragrunt run-all destroy on that dir

Then, in order to remove a dir two PRs are required - first PR to set prevent_destroy = false and destroy the module, and second PR to actually remove the directory.