Using vars from tfvars in locals

[malcolm061990]

Hi. I am using v0.28.18 terragrunt. I want to apply tf module in two different AWS regions. And I want to keep state for each region in different s3 keys like: terragrunt-test/terraform-us-east-1.tfstate terragrunt-test/terraform-us-east-2.tfstate

The only way I thought I could do that was using locals. My configurations:

###### account1.tfvars:
aws_profile = "account1"
aws_region  = "us-east-1"

###### terragrunt.hcl in module dir
include {
  path = find_in_parent_folders()
}

###### terragrunt.hcl in parent dir
# root/terragrunt.hcl
generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  profile = var.aws_profile
  region  = var.aws_region
}
EOF
}

locals {
  aws_region = "${var.aws_region}"
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    bucket         = "${get_env("TG_BUCKET_PREFIX", "tf-bucket")}-${get_aws_account_id()}"
    key            = "${path_relative_to_include()}/terraform-${local.aws_region}.tfstate"
    region         = "us-east-1"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

I run:

terragrunt apply -var-file=account1.tfvars
ERRO[0000] Not all locals could be evaluated:           
ERRO[0000]      - aws_region                                
ERRO[0000] Could not evaluate all locals in block.      
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

If I hardcode aws_region in locals to us-east-1:

locals {
  aws_region = "us-east-1"  
}

and run apply the correct file will be generated:

# Generated by Terragrunt. Sig: nIlQXj57tbuaRZEa
terraform {
  backend "s3" {
    bucket = "tf-bucket-AWS_ACCOUNT_ID"
    key    = "terragrunt-test/terraform-us-east-1.tfstate"
    region = "us-east-1"
  }
}

Question: how can I use the variable from tfvars to generate provider configuration without using any environment vars?

Terraform's locals can work with vars (https://learn.hashicorp.com/tutorials/terraform/locals#use-locals-to-name-resources). So... Please, help.

[ArturChe]

Why do you use var.aws_region for provider generation and local.aws_region for backend generation? It would not works. Use the first approach to let Terraform interpolate variables in generated .tf files from .tfvars. Or use Terragrunt locals, which are available only inside its configuration, but you must put a value (https://terragrunt.gruntwork.io/docs/features/locals/)

[malcolm061990]

Why do you use var.aws_region for provider generation and local.aws_region for backend generation? It would not works. Use the first approach to let Terraform interpolate variables in generated .tf files from .tfvars. Or use Terragrunt locals, which are available only inside its configuration, but you must put a value (https://terragrunt.gruntwork.io/docs/features/locals/)

It doesn't work:

remote_state {
  backend = "s3"
  config = {
    bucket         = "${get_env("TG_BUCKET_PREFIX", "tf-bucket")}-${get_aws_account_id()}"
    key            = "${path_relative_to_include()}/terraform-${var.aws_region}.tfstate"
    region         = "us-east-1"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

I have already mentioned that I tried to use locals. But it doesn't work too. If you know how to get it working please give the example.

[ArturChe]

You are mixing Terraform and Terragrunt locals. You want Terragrunt to interpolate what he can interpolate (like local and path_relative_to_include()) but if a variable needs to be interpolated by Terraform then it needs to be smart enough to skip it (like you do with var.aws_region). Of course, it does not work! Moreover, in Terraform variables are not allowed in the backend config! Use the Terragrunt locals mechanism in your Terragrunt config instead. In your case, it is probably better to have locals in the dependent configs. Read globally defined locals at the end of https://terragrunt.gruntwork.io/docs/features/locals/

[malcolm061990]

You are mixing Terraform and Terragrunt locals. You want Terragrunt to interpolate what he can interpolate (like local and path_relative_to_include()) but if a variable needs to be interpolated by Terraform then it needs to be smart enough to skip it (like you do with var.aws_region). Of course, it does not work! Moreover, in Terraform variables are not allowed in the backend config! Use the Terragrunt locals mechanism in your Terragrunt config instead. In your case, it is probably better to have locals in the dependent configs. Read globally defined locals at the end of https://terragrunt.gruntwork.io/docs/features/locals/

I think we can't understand each other :) I want to run terragrunt apply -var-file=account1.tfvars to have the s3 key for one region and terragrunt apply -var-file=account2.tfvars to have the s3 key for second region. How globally defined locals can help me with that?

[ArturChe]

Wow... Now I got it. And I don't know how to accomplish it without creating subfolders "us-east-1" and "us-east-1". Interesting question!

[malcolm061990]

Wow... Now I got it. And I don't know how to accomplish it without creating subfolders "us-east-1" and "us-east-1". Interesting question!

If you mean creating subfolders in tf module it kills all flexibility of the cool tool terragrunt :(

[yorinasub17]

My guess is that you want to control this using CLI args, and unfortunately, Terragrunt doesn't read the var files that are passed in so it isn't possible doing it using tfvars in the manner you specified.

However, there is an alternative that depends on environment variables that can get close to what you want.

First, convert your terraform tfvars files to use json format. This is necessary so terragrunt can read them in. Then, consider the following terragrunt.hcl file:

###### terragrunt.hcl in parent dir
# root/terragrunt.hcl
generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  profile = var.aws_profile
  region  = var.aws_region
}
EOF
}

locals {
  varfile = get_env("TG_VAR_FILE", null)
  vardata = local.varfile != null ? jsondecode(file(local.varfile)) : { aws_region = "us-east-1" } # some default
  aws_region = local.vardata.aws_region
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    bucket         = "${get_env("TG_BUCKET_PREFIX", "tf-bucket")}-${get_aws_account_id()}"
    key            = "${path_relative_to_include()}/terraform-${local.aws_region}.tfstate"
    region         = "us-east-1"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

terraform {
  extra_arguments {
    commands = get_terraform_commands_that_need_vars()
    arguments = local.varfile != null ? ["-var-file=${local.varfile}"] : []
  }
}

Now you should more or less get what you want if you do:

TG_VAR_FILE=account1.tfvars.json terragrunt apply

[malcolm061990]

My guess is that you want to control this using CLI args, and unfortunately, Terragrunt doesn't read the var files that are passed in so it isn't possible doing it using tfvars in the manner you specified.

However, there is an alternative that depends on environment variables that can get close to what you want.

First, convert your terraform tfvars files to use json format. This is necessary so terragrunt can read them in. Then, consider the following terragrunt.hcl file:

###### terragrunt.hcl in parent dir
# root/terragrunt.hcl
generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  profile = var.aws_profile
  region  = var.aws_region
}
EOF
}

locals {
  varfile = get_env("TG_VAR_FILE", null)
  vardata = local.varfile != null ? jsondecode(file(local.varfile)) : { aws_region = "us-east-1" } # some default
  aws_region = local.vardata.aws_region
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    bucket         = "${get_env("TG_BUCKET_PREFIX", "tf-bucket")}-${get_aws_account_id()}"
    key            = "${path_relative_to_include()}/terraform-${local.aws_region}.tfstate"
    region         = "us-east-1"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

terraform {
  extra_arguments {
    commands = get_terraform_commands_that_need_vars()
    arguments = local.varfile != null ? ["-var-file=${local.varfile}"] : []
  }
}

Now you should more or less get what you want if you do:

TG_VAR_FILE=account1.tfvars.json terragrunt apply

Thanks for the reply! But why is it necessary to use json instead of default tfvars format?

[yorinasub17]

You need to use json so that you can access the variable values within the terragrunt config. read_terragrunt_config does not work with arbitrary HCL (it is only designed to read in HCL in terragrunt.hcl format), so you can't use it to read in the tfvars files to access with local.

[ben851]

I'm with @malcolm061990 on this one - I want to be able to dynamically create infrastructure (in my case on Azure) by simply passing in a new tfvar file. I.e. env1.tfvar, env2.tfvar. It seems odd that a tool designed to keep terraform DRY is unable to parse a tfvar file in its own config. Is this something that would be a welcome addition?

[yorinasub17]

PR to add a helper function to parse and expose tfvars file is certainly welcome!

[malcolm061990]

PR to add a helper function to parse and expose tfvars file is certainly welcome!

What are the next actions?

[yorinasub17]

What are the next actions?

If you are up for implementing a PR, you can go ahead and fork + try to implement this. Otherwise, we will have to wait to see if someone from the community wants to tackle this. Unfortunately, this isn't really a need for us internally at Gruntwork, so it's a bit low on our priority list of terragrunt improvements we would take on internally - community PR is the best way this gets implemented.

[kartikay101]

Hey I am looking to pick up this up but I saw the prs-welcome was removed from the issue.

Wondering if this is still a valid issue ?

[yorinasub17]

@kartikay101 this is still an issue and a PR is welcome! We recently removed all the prs-welcome and help-wanted labels because all issues are welcome for contribution. The only note to add is that anything labeled high-priority is something that we prioritized internally at Gruntwork so there is a good chance someone is working on it, so you will want to check with us on the issue before starting work.

[kartikay101]

Hey been a little busy with things here, finally got some time to look into this.

Thinking about adding something like read_tfvars that can accept a file stream or file path I think and then return a json which can be used here.

Looking into the parsing bit, is there already something in place ? I was looking to find/use something created by terraform which they might be using.

[lemoo5sg]

Hey, sorry to ask, is read_tfvars still possible ? or is there a similar / an available alternative created meanwhile? Without this feature, we have to duplicate our vars in a tfvar and in a yaml file.

[alikhil]

Hi! Is there anyone who is working on this issue? I am about to contribute read_tfvars implementation

[denis256]

Included in release https://github.com/gruntwork-io/terragrunt/releases/tag/v0.52.5