Terraform metadata available for interpolation

mwarkentin commented 4 years ago

Use-cases

As discussed with @danieldreier in Hangops #terraform channel:

We would like to be able to make it easier to "backtrace" terraform-managed resources from AWS using tags. We currently try to have a minimum of tagging resources managed by Terraform with Terraform = True or Automation = Terraform so that we're aware whether something we're looking at in AWS (console, CLI, etc) is managed by Terraform (or Cloudformation, manually created).

It would be nice if we could go further and tag resources with more metadata in order to make it easier to get back to the code which manages those resources. Some examples of metadata which would help:

Terraform state location
Terraform module name
Terraform module version
Terraform filename (foo.tf) and line number defining the resource

Attempted Solutions

We've done some of this using manual tags, but it's pretty clunky. To have any idea where a module is being instantiated requires setting up variables on the module and passing those in during instantiation, hardcoding path strings (which could change if we move the code to another file), etc.

Proposal

There are a few things that could make this easier:

Providing some of these internal metadata as interpolations or pre-filled variables that we could implement manually
Provide some way to tell Terraform to automatically tag all resources with certain metadata if available

References

My second proposal sounds somewhat related to the existing "default tags" issues
- https://github.com/terraform-providers/terraform-provider-aws/issues/7926
- https://github.com/hashicorp/terraform/issues/2283

apparentlymart commented 4 years ago

Thanks for recording these use-cases, @mwarkentin.

There is a tension here in that Terraform's design for modules is attempting to minimize situations where a module's behavior may vary depending on characteristics other than its input variables, because that could make it much harder to predict the impact of a particular change. This is, for example, why the local name that the caller chose for a module is not available within the module itself: that name is supposed to be entirely under the control of the caller, so it would be very surprising if the called module were to make use of that name in other ways.

The subset of this metadata that relates to information within the module itself -- for example, a relative path from the module's package directory to the file that defined a particular resource -- could potentially avoid that pitfall, but it would still create a new possibility where a seemingly-unrelated change elsewhere in a file could affect the configuration of some other resource, by changing the line number where it was defined.

This comment is not meant to suggest that what you're requesting is impossible, but we do need to tread very carefully here because this sort of functionality could potentially lead to Terraform modules being considerably more "brittle" than they are today, by breaking their encapsulation.

mwarkentin commented 4 years ago

@apparentlymart thanks for the explanation. I figured it would be something like that. Can the caller of a module reference it's own configuration?

eg. if we had a module with a variable called module_name and module.foo - could module.foo reference itself somehow to pass in that variable? Just thinking outloud.. this would still be a decent amount of wiring to get everything working together but maybe retains the encapsulation that you're looking for?

There was a comment in one of the linked PRs talking about the CDK and the way it does recursive tagging - maybe this would be another way? If we could "tag" the module itself which would handle tagging the resources within?

Thanks again!

mwarkentin commented 3 years ago

@apparentlymart this project launched recently which does a lot of what I wanted (although I'm still trying to figure out if/how modules are supported): https://yor.io/

Some examples of the auto-tagging they added:

yor_trace = "724cd8ac-5d73-4887-92cd-23c6c58d66a8"
git_org = "bridgecrewio"
git_repo = "terragoat"
git_file = "README.md" # This is the path from the repo root dir.
git_commit = "47accf06f13b503f3bab06fed7860e72f7523cac" # This is the latest commit for this resource.
git_last_modified_At = "2020-03-28 21:42:46 +0000 UTC"
git_last_modified_by = "schosterbarak@gmail.com"
git_modifiers = "schosterbarak/baraks" # These are extracted from emails (everything before the @ sign). This can also be done 
for the git_last_modified_by tag.

hashicorp / terraform