Named resource groups/namespaces

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

Other

42.14k stars 9.47k forks source link

Terraform Version

1.5

Use Cases

Enabling grouping related elements in modules together under a given namespace to allow simplifying names.

Many cases exist where we have some kind of repeated IaC, but the overhead of using modules for them increases the complexity by needing variables and outputs to be redeclared.

An example would be setting up an AWS VPC. You might need to declare VPC endpoints for various AWS services, each needing a security group, security group rules, and other possible configurations like IAM policies. This quickly becomes very hard to separate concerns where the configuration varies slightly each time.

Having the ability to group resources into namespaces would make this easier to read since IDEs could collapse/fold entire namespaces at a time, and all resources under the namespace are enforced to have a consistent base name.

Attempted Solutions

Right now, we can use modules to physically separate groups of resources but this introduces a lot of repeated code by having to redeclare variables and outputs.

We could also just use more complex names but this becomes harder to read and is difficult to enforce consistently.

Proposal

Allow using an HCL named block at the top level of a Terraform source file to declare a namespace.

namespace "stubs" {
}

The namespace can then hold any modules, datasources, resources, etc that relate to the namespace.

Resources within the namespace can refer to each other using their normal names like in existing Terraform code. This can lead to them taking more generalised names that are clearer to their relative module. Take this theoretical example:

/**
 * Resources related to the creation of a stub API server.
 */
namespace "wiremock_stub" {
  resource "wiremock_image" "image" {
    version = "1.2.3"
  }

  resource "wiremock_server" "server" {
    image         = wiremock_image.image.id
    stub_mappings = wiremock_mappings.mappings.json
  }

  data "wiremock_mappings" "mappings" {
    ...
  }
}

Resources can be interacted with outside their namespace using their fully qualified name.

namespace "wiremock_stub" {
  ...

  resource "wiremock_server" "this" {
    ...
  }

  ...
}

resource "something_else" "whatever" {
  stub_server_endpoint = wiremock_stub.wiremock_server.this.url
}

This could even be extended further to allow features such as those that I suggested in https://github.com/hashicorp/terraform/issues/33750 to be considered, like allowing blocks to be optional in bulk.

References

No response

Hi @ascopes! Thanks for this proposal.

The way I understand what you are proposing is that a namespace is essentially an "inline module", with a closure-like capability to capture symbols from the surrounding lexical scope.

This does seem like a plausible idea in principle, but challenging to design in the details:

Terraform currently has no situations where one scope can inherit from another like this. I expect it would require an entirely new design for expression evaluation to be able to take into account there being objects from multiple namespaces in scope at the same time.
Because of the resource_type.resource_name syntax for referring to a resource, it isn't generally possible for us to add new kinds of top-level symbol without breaking compatibility with providers that might have conflicting resource types. However, we could potentially mitigate that by calling your namespaces "inline modules" and reusing the module.foo syntax to refer to them, since that's already reserved as a special case. (This means it would not be possible to have both a normal module and an inline module of the same name in the same scope, but that seems fine to me.)
Terraform's graph builder currently only supports each node providing symbols to a single module scope, so that subsystem would need considerable rethink to allow an object in one module to directly depend on another without an intermediate "bridging object" like an input variable or output value. (Or maybe this would work by quietly inserting anonymous extra graph nodes that bridge across that gap, as a hidden implementation detail.)
All resources do still need to have a unique address within the configuration for tracking across runs in the state, so we'd need to figure out what those would look like. Again, if we consider these as "inline modules" then we can potentially reuse the existing addressing scheme for objects inside modules.

With all of that said, my best idea for how to do this right now would be to treat a "namespace" as a funny kind of module which gets different treatment in the graph builder and expression evaluator so that the propagation of symbols between scopes happens automatically via hidden proxy nodes that serve a similar role as input variables and output values by transferring dependencies between the separate module namespaces.

That's far easier said than done, so I think this would require considerably more design work before we could say whether it's feasible and whether it would meet the use-cases you presented, but we will leave this issue open to remember the use-case and initial ideas.

Thanks again!

hashicorp / terraform