Using Output From Module A as Input into Module B

[hammadzz]

I have tried this pattern but seem to encounter an issue trying to use an output from a module as an input for another module in terraform.tfvars.

For example in the db terraform.tfvars, using this:

db_subnet_group_name          = "${module.vpc.db_subnet_group_name}"

throws the error: Underlying error: Invalid interpolation syntax. Expected syntax of the form '${function_name()}', but got '${module.vpc.db_subnet_group_name}'

[brikis98]

You don't put that code in terraform.tfvars. That file is just for setting plain values for Terraform values. If you want to reference another module, you put that code in your .tf files.

[gevial]

Hi @brikis98

Sorry - does it mean that managing environments with terragrunt.hcl as described in https://github.com/gruntwork-io/terragrunt#keep-your-terraform-code-dry does not allow passing outputs of one module as inputs to another?

I mean, terragrunt.hcl doesn't support variables, only plain values... what would be the idiomatic way to allow modules exchange data with terragrunt?

Thanks

[yorinasub17]

Unfortunately, currently terragrunt does not have a way to read in outputs from other modules into the inputs in the terragrunt.hcl file.

An alternative way to handle message passing between modules is to use the terraform_remote_state data source. If you are using this example, you should be able to infer the state file path in the bucket because it matches the file path of your live folder. For example, the path to the mysql state file in the prod environment is us-east-1/prod/mysql/terraform.tfstate.

In our environment, we use a global varfile (automatically included by terragrunt) that sets the bucket and region of the remote state as variables, that is then used in the submodules to setup the terraform_remote_state data source. The key is then inferred based on environment and region of the module call.

[gevial]

Yes, thanks, I've finished up with the approach described here: https://github.com/gruntwork-io/terragrunt/issues/303#issuecomment-510309532

In short: you define root variables in parent terragrunt.hcl and folder-specific in its own terragrunt.hcl. Intermediary vars (e.g. region-wide or env-wide) are stored in YAML and included as suggested above.

If a folder configuration doesn't require outputs from another modules, I include it with terraform {} block in terragrunt.hcl right away. Otherwise, I use terragrunt.hcl together with main.tf.

Thank everybody.

[gevial]

The only downside of the approach above though is that I have to declare all variables used by modules in main.tf.

Consider the following structure:

.
├── dev
│   ├── eks
│   │   ├── main.tf
│   │   └── terragrunt.hcl
│   ├── rds
│   │   └── terragrunt.hcl
│   ├── vars.yaml
│   └── vpc
│       └── terragrunt.hcl
├── prod
│   └── terragrunt.hcl
└── terragrunt.hcl

Consider dev/eks for example. As this configuration depends on dev/vpc outputs, I cannot use terragrunt.hcl all alone (the support for get_output() would fix that by the way). So I have to have main.tf which reads from remote state and use our internal module.

The thing is that I have to declare variables in dev/eks/main.tf like this:

terraform {
  backend "s3" {}
}

variable "cluster_name" {}
variable "aws_region" {}
variable "kubernetes_version" {}
variable "instance_type" {}
variable "remote_state_bucket" {}

data "terraform_remote_state" "vpc" {
  backend = "s3"
  config = {
    bucket = var.remote_state_bucket
    key    = "dev/vpc/terraform.tfstate"
    region = var.aws_region
  }
}

module "eks_cluster" {
  source              = "/Users/user/r/tf-modules//eks_cluster"
  cluster_name        = var.cluster_name
  aws_region          = var.aws_region
  kubernetes_version  = var.kubernetes_version
  instance_type       = var.instance_type
  vpc_id              = data.terraform_remote_state.vpc.outputs.vpc_id
  vpc_private_subnets = data.terraform_remote_state.vpc.outputs.private_subnets
}

It'd be really appreciated if someone can suggest another more clear way to pass variables. Thanks.

[gevial]

If I don't declare variables, I get multiple errors:

Error: Reference to undeclared input variable

  on main.tf line 25, in module "eks_cluster":
  25:   instance_type       = var.instance_type

[gevial]

Well, it seems I've found a better workaround. I've finally managed to define all configurations with a single terragrunt.hcl file in each while using each other module's outputs.

Here's how terragrunt.hcl of the dependant configuration looks like:

include {
  path = find_in_parent_folders()
}

terraform {
  source = "${get_parent_terragrunt_dir()}/../tf-modules//eks_cluster"
}

inputs = merge(
  yamldecode(file("${get_terragrunt_dir()}/${find_in_parent_folders("vars.yaml")}")),
  {
    kubernetes_version  = "1.13",
    instance_type       = "t3.small",
    vpc_id              = trimspace(run_cmd("terragrunt", "output", "vpc_id", "--terragrunt-working-dir", "../vpc"))
    vpc_private_subnets = run_cmd("terragrunt", "output", "private_subnets", "--terragrunt-working-dir", "../vpc")
  }
)

dependencies {
  paths = ["../vpc"]
}

[brikis98]

@gevial Those are some very creative workarounds!

Do you think it would make sense to add dedicated helpers for some of that? E.g., For reading input variables from an external file (e.g., a get_input helper)? Or running terragrunt output to get an output from another module (e.g., a get_output helper)? The latter in particular could also allow Terragrunt to track dependencies much better between modules.

If those sound worth adding, anyone up for a PR? Adding helpers to Terragrunt is much easier these days... You just add a function here: https://github.com/gruntwork-io/terragrunt/blob/master/config/config_helpers.go#L71-L84

[gevial]

Thanks @brikis98

At this stage I think both helpers would be helpful. However going forward I think the best solution would be to consider a more holistic approach. One of the good examples is Hiera - Puppet's built-in tool to manage hierarchy of configuration parameters from various sources. In Terragrunt world this kind of solution might support registering outputs as possible source as well.

I will try to devote some time to get_output. Please let me know what's the best approach to call Terragrunt from within Terragrunt - is it cli.CreateTerragruntCli() and then run()? Thanks

[brikis98]

One of the good examples is Hiera - Puppet's built-in tool to manage hierarchy of configuration parameters from various sources. In Terragrunt world this kind of solution might support registering outputs as possible source as well.

Could you give an example of what this would look like?

I will try to devote some time to get_output. Please let me know what's the best approach to call Terragrunt from within Terragrunt - is it cli.CreateTerragruntCli() and then run()? Thanks

Probably this method: https://github.com/gruntwork-io/terragrunt/blob/master/cli/cli_app.go#L234

[gevial]

Let's say we have a tf-live repo like this https://github.com/gruntwork-io/terragrunt-infrastructure-live-example You define a hierarchy of data lookups in root terragrunt.hcl and then when you run Terragrunt in any sub-folder like prod/us-east-1/prod/mysql Terragrunt already knows where to query for input variables and the most specific value it finds (another feature is to merge values found on different hierarchy layers). The idea is explained here: https://puppet.com/docs/puppet/6.6/hiera_intro.html

@brikis98 Looks like I can't call cli.RunTerragrunt() from config/config_helpers.go as it will introduce a cyclic dependency: config -> cli -> config Implementing get_output() as shell command run feels a bit hacky and rearranging packages is too much of a change for the single helper function in my opinion. What do you think?

[brikis98]

You define a hierarchy of data lookups

I'm curious what that would look like in the Terragrunt world?

Looks like I can't call cli.RunTerragrunt() from config/config_helpers.go as it will introduce a cyclic dependency: config -> cli -> config. Implementing get_output() as shell command run feels a bit hacky and rearranging packages is too much of a change for the single helper function in my opinion.

Moving RunTerragrunt and its downstream dependencies to a new package (e.g., run) makes sense. Looking at it now, it never really belonged in cli to begin with, and has made the cli package quite bloated. Agreed it's a lot of work for a single helper function, but it will likely be a useful change in general.

[LinguineCode]

Looks like this comment thread has gone off topic, although I am a huge fan of Hiera (as in the popular Puppet tool) and can't wait to see it included with Terraform one of these days

Responding to the original question: I use SSM Parameter Store (I exclusively use AWS). The module that creates the infrastructure (e.g. vpc subnets) also creates aws_ssm_parameter. Then, when another module needs the subnet-id's, it will do a data.aws_ssm_parameter lookup to get it. I avoid using data lookups from data.terraform_remote_state or module outputs {}

[taylorturner]

I'm really disappointed Terragrunt doesn't support using one module's output as the input for another module. I'd think this would be a pretty fundamental feature when using Terraform modules.

[yorinasub17]

Oh it looks like we forgot to close this issue, as this has been supported for a while with dependency blocks:

https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#dependency