Create a Dependency Injection provider

[pauldraper]

Current Terraform Version

0.12.13

Use-cases

There are often global-ish resources, whether by necessity or preference.

• AWS API Gateway requires account-level configuration to do logging. There can sensibly be only one aws_api_gateway_account resource per AWS account.
• Datadog can replicate AWS Cloudwatch Logs via a Lambda function. Ideally, there would be only one aws_lambda_function resource, and each part of infrastructure with a log group would register a log listener and lambda permission. However, since logging is a universal concern, it must be accessible very many places.

Attempted Solutions

The obvious solution is plumbing everything though. This however is tedious and lack abstraction.

A generic "context" variable could be passed through to every resource. While providing more abstraction, this is likewise verbose.

Global static data can be handled by re-creating a module each time it is needed; however, that doesn't work for actual resouces.

Proposal

In other programming languages, this problem is often solved by dependency injection containers.

Terraform's provider infrastructure already does this, e.g. it is not required to specify AWS region for every resource.

The only thing really missing currently is the ability to define injection keys in .tf files.

local_provider "custom_thing" {
  value = resource.config_resource.attr
}

local_provider "custom_thing" {
  alias = "second"
  value = "my_value2"
}

resource "a_resource" {
  a_attribute = custom_thing.default # reference it in any descendant resource
}

module "a_module" {
  # implicity passed to module
}

module "a_module2" {
  providers = { custom_thing = custom_thing.second } # override
}

References

[teamterraform]

Hi @pauldraper! Thanks for sharing this feature request.

The existing pattern for dependency injection in Terraform is module composition. This allows configuration authors to apply dependency inversion principles by using references in the root module to connect child modules acting as services with child modules serving as clients. In this model, the root module serves as the injector, and interfaces are defined by declaring output values on service modules and input variables on client modules.

In your example, it sounds like "datadog log transport" is a service which is consumed by various clients. That suggests that datadog log transport would be a module whose outputs form the interface of this "log transport" abstraction, and thus the object representing the module can be passed into other modules that will consume that service:

module "datadog_logging" {
  source = "./modules/datadog_logging"

  # (any necessary arguments)
}

module "uses_datadog_logging" {
  source = "./modules/uses_datadog_logging"

  datadog_logging = module.datadog_logging
}

We're not familiar enough with datadog's approach to log transport to know if it's possible to create a vendor-agnostic abstraction around it, but if there were conceivably some generic interface that could bridge from Cloudwatch Logs to multiple log aggregation vendors (perhaps the commonality is providing a Lambda function whose input arguments are what Cloudwatch Logs produces) then you could design a general interface for logging connectors such that the clients can accept outputs from any module that conforms to that general interface. That general technique is discussed in the Multi-cloud Abstractions section of the Module Composition guide.

As your infrastructure grows you will likely wish to decompose it into multiple separate configurations that can be applied separately and pass data via data sources. In that case, you can build a data-only module that produces the same set of outputs as some corresponding management module but that uses some mechanism to retrieve an existing object created elsewhere, and then swap out the management module for the data-only module without the modules representing clients of that service needing to change.

With that said, we're not sure we fully understood what you were suggesting here, since in your example you seem to be defining something analogous to a Terraform provider configuration, and we're not sure how that fits in with the idea of dependency inversion. Could you share a little more detail, and perhaps describe the ways you see what you are imagining as different from the existing pattern described above?

Thanks again!

[pauldraper]

and thus the object representing the module can be passed into other modules that will consume that service:

I describe this in original post: "The obvious solution is plumbing everything though. This however is tedious and lack abstraction."

In a technical sense "DI" just means "accepting parameters", and that is exactly what you can do today; however I am wanting "implicit DI".

Note that Terraform does not require you to pass the aws provider, gcp provider, azure provider datadog provider, etc. through each module and resource. While that would have the advantages of explicitness, it would unduly burden the user for such a basic object used in so many places.

Instead, the provider can automatically cascade down through dozens of nested modules and hundreds of resources. If customization is required, at any point in the module/resource tree, a user can optionally specify a different provider for that portion of the tree. While this somewhat "magic", I believe the ergonomics of providers are superior to an explicit approach, and the Terraform authors evidently agree.

Now imagine that a user wants similar behavior for something equality ubiquitous in his infrastructure (say, a logger Lambda), but preferable without having to write and compile his own provider. I want to have the same implicitness as a provider, but for any Terraform information.

[franz-josef-kaiser]

This.