(feat): introduce Terragrunt schema

[kvendingoldo]

Imagine that we have the following structure that we will create via terragrunt apply-all.

.
├── non-prod
│   ├── account.hcl
│   └── eu-north-1
│       ├── region.hcl
│       └── stage
│           ├── eks
│           │   └── terragrunt.hcl
│           ├── env.hcl
│           └── redshift
│               └── terragrunt.hcl
└── terragrunt.hcl

If I delete redshift folder and make terragrunt apply-all again. Redshift won't be deleted. I suggest to introduce folder schema that will allow to delete modules that was deleted from Terragrunt configuration. In case if folder does not exist anymore but exist in schema we can download state and make terrafrom destroy. It will guarantee to delete old resources.

[kvendingoldo]

@brikis98 @yorinasub17 WDYT?

[yorinasub17]

As mentioned in https://github.com/gruntwork-io/terragrunt/issues/847#issuecomment-615374032, this is not really a direction we want to head towards, as there are many complications to think through. Here are a few that quickly pop to mind:

• Where would this schema be tracked/stored? How would it work in a team setting (remote state config vs local state)?
• What should happen for modules that had prevent_destroy = true? Should it track that in this schema, and if so, should it ignore those until an apply is run that unsets this like force_destroy on aws_s3_bucket?
• Doing the destroy as part of apply-all does not seem like a friendly UX and deviates from the model of terragrunt wrapping terraform commands (e.g., what if someone just wants to run plain old apply-all without any destroy features?). It feels like we should have a new command for this purpose.
• What happens if you move folders? Should it do a destroy and an apply? Or should it intelligently shift state as mentioned in the thread linked above?
• What about versioning? How do we handle schema changes across terragrunt versions? Should it automigrate like terraform?

Also mentioned in the comment, would be nice to see a detailed proposal that walks through a concrete UX flow of the feature including a sketch of how the file works (e.g., what kind of data would it store?), where it is stored and how that is configured, and some thought experiments of how it will handle certain team scenarios like merge conflicts and concurrency. If there is a way to work through these issues in a maintainable and easy to understand way (e.g., having two state files to keep track of for each module can be confusing), we would be open to implement this.

[lorengordon]

You can kind of get a workaround by using raw terraform to call the modules, in addition to terragrunt, though you lose a lot of the benefits of terragrunt and need to redefine variables and outputs. :(

see changes to redshift directory:

.
├── non-prod
│   ├── account.hcl
│   └── eu-north-1
│       ├── region.hcl
│       └── stage
│           ├── eks
│           │   └── terragrunt.hcl
│           ├── env.hcl
│           └── redshift
│               └── terragrunt.hcl
│               └── main.tf
│               └── variables.tf
│               └── outputs.tf
└── terragrunt.hcl

redshift/terragrunt.hcl:

terraform {
  source = "./"
}

redshift/main.tf:

module redshift {
  source = // url you _would_ have put in terragrunt.hcl

  // pass-thru variable inputs

Then instead of deleting the whole directory, you just delete module from redshift/main.tf...