Allow to interpolate ${var.*} inside backend configuration

[alekbarszczewski]

Terraform Version

Terraform v0.11.3
+ provider.aws v1.8.0

Use case

For now I am using local (file) state. I have stage variable that defaults to "dev", local project variable that is a constant "my-project" and local prefix variable that is equal to "${local.project}-${var.stage}". I prepend this prefix to all resources, for example I have bucket my-project-dev-files, dynamo table my-project-dev-users and so on. This way I can keep separate projects and stages (environments like dev/prod/etc) in one AWS account. When deploying my project from travis for example I just pass stage variable via env to terraform (var is based on repository branch). So develop branch uses stage "dev", master uses "prod" and so on.

Now I want to use remote state via S3 backend. However I have problem because terraform does not allow interpolation when configuring backend. I would like to use two separate S3 buckets for dev and prod stages for example. I know that I can pass some config via terraform argument -backend-config="KEY=VALUE", but it requires me to pass this additionally to stage var: TF_VAR_stage=dev terraform apply -backend-config="bucket=my-project-dev". It would be nice to be able to still interpolate at least ${var.*} inside backend.

Or maybe my approach is not the right one?

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "${local.prefix}-state"
    key    = "state.tfstate"
  }
}

data "terraform_remote_state" "network" {
  backend = "s3"
  config {
    bucket = "${local.prefix}-state"
    key    = "state.tfstate"
  }
}

provider "aws" {
  region  = "us-east-1"
}

data "aws_region" "current" {
  current = true
}

variable "stage" {
  type = "string"
  default = "dev"
}

locals {
  project = "my-project"
  prefix = "${local.project}-${var.stage}"
}

[apparentlymart]

Hi @alekbarszczewski! Thanks for this question / feature request.

Backends are specified the way they are because any operation Terraform does starts by first accessing the backend, before doing any other work. For example, the terraform workspace list command requires the backend to be able to inspect remote data to determine which workspaces exist, which for the S3 backend is currently implemented by prefix-matching keys within the given S3 bucket.

Our current recommendation is to treat Terraform -- and thus the Terraform states -- as something "outside" the environments they manage, rather than as part of the environment. In practice, that means that e.g. there is an S3 bucket for your Terraform states that is separate from the environments managed by Terraform. Some organizations think of this as a "special" environment called e.g. admin or tools, using this as a home for any pan-environment management tools, e.g. also including a CI system that is used to run Terraform.

We've documented an example pattern for Terraform across multiple AWS accounts, which is more separation than you want to do here but can be simplified down to an equivalent idea with multiple resource sets under a single AWS account.

---

In future we could add a feature to the S3 backend in particular so that it's able to alternatively prefix-match (or, perhaps, suffix-match) the bucket name itself, allowing you to have multiple S3 buckets in your account like mycompany-terraform-production, mycompany-terraform-staging, and then list all of the buckets in your account to find which workspaces exist.

So far it's seemed more common for organizations to use a separate AWS account per environment, which led to us creating the document above, but if multiple buckets in a single organization is also a common pattern it certainly seems technically possible to add it, at the expense of requiring some more liberal IAM policies to enable the bucket names to be enumerated. I'd be interested to hear if you and others would find such a feature useful, and if so what sort of bucket naming conventions you'd want to support. (You already shared this, but I'd like to see what patterns others follow too, in order to inform the requirements for the feature.)

[alekbarszczewski]

@apparentlymart Thanks for detailed response. I somehow missed the fact that in remote state stored in S3 each state (per workspace) is prefixed with <workspace>:. So I ended up having single state bucket for both staging/prod workspaces and it works perfectly. I just prefix all resources (their unique names in aws) with my-project-${terraform.workspace} instead of using variable my-project-${var.stage}. For me this is enough when keeping multiple environments (workspaces) on single AWS account.

But I agree that keeping each environment on separate AWS account is the best approach since it separates everything and you don't have to prefix each resource name.

---

I ended up using following .travis.yml file for continious integration / tests of my project:

language: node_js
node_js:
  - '8'
env:
  global:
    - AWS_ACCESS_KEY_ID=***
    - AWS_AWS_DEFAULT_REGION=us-east-1
    - secure: "***"
    - TEMP_WORKSPACE="travis-build-$TRAVIS_BUILD_ID"
install:
  - yarn
cache:
  directories:
    - node_modules
before_script:
  - echo "PREPARING TEMP WORKSPACE $TEMP_WORKSPACE"
  - ./bin/terraform init aws-stack/ &&
    ./bin/terraform workspace new "$TEMP_WORKSPACE" aws-stack/ &&
    make package &&
  - ./bin/terraform apply --auto-approve aws-stack/
script:
  - ./bin/terraform output aws-stack/
after_script:
  - ./bin/terraform destroy -force aws-stack/ &&
    ./bin/terraform workspace select default aws-stack/ &&
    ./bin/terraform workspace delete "$TEMP_WORKSPACE" aws-stack/

In before_script I create new workspace based on travis build id and deploy stack.
In script I will run my integration tests, based on terraform output.
In after_script (in case of both success/failure of script) I cleanup everything: destroy stack and delete workspace.

---

Unfortunately I ran into another issue: https://github.com/hashicorp/terraform/issues/17300

[alekbarszczewski]

@apparentlymart Hey, after rethinking this issue is a blocker for me. The idea of two AWS accounts per environment (dev/prod) is great, but right now, without interpolation of S3 bucket in backend it's not possible to separate dev and prod environments completely. There always must be a shared S3 bucket for remote state for both environments.

What we want now is to make environments completely separate from each other for security reasons/ I think there is no point of not allowing them to be separate. Both envs does not have to know about each other, share state or anything. The only thing that is blocking this is remote state s3 bucket that can't be interpolated in config and has to be shared because bucket names are globally unique.

I read docs about using three AWS accounts and role delegation but this is not what we want. For security reasons we want to make both environments completely separate.

Terraform is great and we are using it for our second project in production, however this one is really blocker for us. Right now we are considering generating backend.tf file dynamically when deploying to one of the environments, but it's not a way to go...

By any chance will you improve/fix this in the nearest future? Either interpolating backend bucket name or at least some dynamic prefix for bucket or something like that?

[arocki7]

We are having the same issue here.

[UncleTawnos]

simply for simplicity of configuration I'd LOVE to see it hapening

[apparentlymart]

The current way to achieve this entirely separate configuration per environment is to not use "workspaces" at all and instead use multiple separate backend configurations.

To do this, the backend block in the main configuration would be empty:

terraform {
  backend "s3" {
  }
}

Then create another file in the directory for each environment. These can either be hand written and checked in or generated by your automation wrapper, depending on what makes sense for your use-case. For example, here's a production.tfbackend file based on the original comment:

  bucket = "example-production-state"
  key    = "state.tfstate"

Switching between these environments is then done by reconfiguring the backend itself, rather than switching workspaces:

terraform init -reconfigure -backend-config=production.tfbackend

I understand that right now this means that you also need to pass a similar or identical value into a variable, which is inconvenient.

The backend and workspace features will get an overhaul in a future release to better permit separate configuration per workspace. The configuration can never support interpolation for the reasons I stated before, but a per-workspace configuration capability would allow the total separation between environments that you are looking for. The current focus of the Terraform team at HashiCorp is on the 0.12 release (which does not include backend changes) but we intend to address this in a later release.

[pixelicous]

@apparentlymart I am not entirely sure i am answering the same thing, but just to give my two cents here for anyone going through this post, it is possible to use workspaces in such/similar fashion. We have a folder structure that resembles provider>account>environment>appmodule We use workspaces as our differentiator for running the same module folder but on a different region. In my opinion the resources you would like to provision per region most likely would be the same, but it aint so true per environment, anyhow.. we can then use terraform.workspace as our location anywhere as well.

Any module ("appmodule" folder above) have a backend configuration but just the type is set, the wrapper always runs init first and the backend variables are passed to terraform at runtime (aka partial configuration). After init whenever plan/apply runs and we use remote state with interpolation it look as follows:

data "terraform_remote_state" "my_remote_state" {
  backend = "azurerm"
  workspace = "${terraform.workspace}"
    config {
    storage_account_name = "xxxxxx"
    container_name       = "${var.environment}"
    key                  = "${var.environment}-network.tfstate"
    access_key = "${var.access_key}"
  }
}

[omar]

The solution proposed by apparentlymart requires our build system to output files which is something I'd prefer to avoid to keep inline with how other build systems are generally configured (i.e. through environment variables). Additionally, I'd like to maintain parity between how our CI/CD systems run and how developers run things locally eliminating the need for command line switches when running specific commands.

If the backends supported pulling their values from environment variables in the same way they pull from the command line argument, interpolation could be avoided. For example:

• backend.tf:

  terraform {
  backend "s3" {
  # Keep this empty
  }
  }

• Environment Variables:

export TF_VAR_stage="production"
export TF_WORKSPACE="production"
export TF_BACKEND_S3_BUCKET="production-state"
export TF_BACKEND_S3_KEY="state.tfstate"

CI/CD systems and developers just need to configure the proper environment variables when running and everything will run the same. No need for extra command line switches when running Terraform commands.

If there's enough support for this idea, I could see myself contributing it. But I want to get a TF team member's take on this in case I'm missing some important context.

[LinguineCode]

Good to see a fresh issue created on this limitation. @omar's suggestion is, obviously, better than having a wrapper script that creates state.tf files prior to terraform being run. Show support to his idea by reacting with 👍 on https://github.com/hashicorp/terraform/issues/17288#issuecomment-462899292

[omar]

I've opened a PR, see #20428. Feedback welcome.

[manniche]

@alekbarszczewski The PR #20428 has been closed with information about how to accomplish the objective that was the reason for opening this issue. Should this issue be closed?

[crizstian]

For Consul backend the environment variables almost works but still is prompting for consul path

this is my provider.tf

provider "consul" {
  address    = var.consul_address
  datacenter = var.consul_datacenter
  scheme     = var.consul_scheme
}

provider "vault" {}

terraform {
  backend "consul" {}
}

this are my envs

# General vars
export HOST_IP="172.20.20.11"
export PRIMARY_HOST_IP="172.20.20.11"
export DATACENTER=dc1
export PRIMARY_DATACENTER=dc1
export SECONDARY_DATACENTER=dc2

# Vault env vars
export VAULT_CACERT=/var/vault/config/ca.crt.pem
export VAULT_CLIENT_CERT=/var/vault/config/server.crt.pem
export VAULT_CLIENT_KEY=/var/vault/config/server.key.pem
export VAULT_ADDR=https://${HOST_IP}:8200

# Nomad env vars
export NOMAD_ADDR=https://${HOST_IP}:4646
export NOMAD_CACERT=/var/vault/config/ca.crt.pem
export NOMAD_CLIENT_CERT=/var/vault/config/server.crt.pem
export NOMAD_CLIENT_KEY=/var/vault/config/server.key.pem

# Consul env vars
export CONSUL_SCHEME=https
export CONSUL_PORT=8500
export CONSUL_HTTP_ADDR=${CONSUL_SCHEME}://${HOST_IP}:${CONSUL_PORT}
export CONSUL_CACERT=/var/vault/config/ca.crt.pem
export CONSUL_CLIENT_CERT=/var/vault/config/server.crt.pem
export CONSUL_CLIENT_KEY=/var/vault/config/server.key.pem
export CONSUL_HTTP_SSL=true
export CONSUL_ENCRYPT_KEY="apEfb4TxRk3zGtrxxAjIkwUOgnVkaD88uFyMGHqKjIw="
export CONSUL_SSL=true

# Terraform env vars
export TF_VAR_consul_address=${HOST_IP}:${CONSUL_PORT}
export TF_VAR_consul_scheme=${CONSUL_SCHEME}
export TF_VAR_consul_datacenter=${DATACENTER}
export TF_BACKEND_CONSUL_ADDRESS=${TF_BACKEND_CONSUL_ADDRESS}
export TF_BACKEND_CONSUL_PATH="terraform/${DATACENTER}.state"
export TF_WORKSPACE="production"

this is tf output

root@dc1-consul-server:/vagrant/provision/terraform/tf_cluster/primary# terraform init
Initializing modules...
- tf_consul in ../../consul
- tf_consul.acl in ../../consul/acl
- tf_consul.intentions in ../../consul/intentions
- tf_consul.kv in ../../consul/kv
- tf_consul.prepared-queries in ../../consul/prepared-queries
- tf_vault in ../../vault
- tf_vault.auth-methods in ../../vault/auth-methods
- tf_vault.generic-endpoints in ../../vault/generic-endpoints
- tf_vault.generic-endpoints.app-secrets in ../../vault/generic-endpoints/app-secrets
- tf_vault.generic-endpoints.consul in ../../vault/generic-endpoints/consul-secrets
- tf_vault.generic-endpoints.userpass in ../../vault/generic-endpoints/userpass
- tf_vault.policies in ../../vault/policies
- tf_vault.secrets in ../../vault/secrets

Initializing the backend...
path
  Path to store state in Consul

  Enter a value:

[throwaway8787]

Running interpolation on the backend, then pulling the backend, then running interpolation for the remaining items shouldn't be an impossible thing to ask for.

Why is it like pulling teeth with hashicorp to get the usage patterns everyone wants, supported?

[red8888]

just curious where this is on the road map

I have lots of folders/workspaces and I just want to be able to dynamically set the state file path (im using s3) via the relative path of the folder.

We run terraform with atlantis (so like CI we can have it set env vars) but we also need to be able to plan locally. Making users set env vars for each workspace when planning or writing some wrapper that does that and executes terraform is really wonky. Also not as safe. I like having the backend hardcoded into the config.

Getting the relative path with tf functions is easy but of course interpolation isnt allowed.

I dont want to restore to terragrunt for this but this is one reason why people are still using it. I know they are a hashicorp partner but im pretty sure internally hashicorp folks dont love how many of their users are relying on it.

[DeshKancharla]

Hi all, I tried @apparentlymart's, solution and it works for our environment. However, I wanted to know how can supply more than one backend config file to a terraform module. For example, I have a module that stores its state in a state file in s3bucket xyz but reads some values from another state file(as data source) in another bucket abc. I am trying to understand how can I tell terraform to use one config files for storing its state and another config file to read data from?

[AlexGoris-KasparSolutions]

I'd like to add another use case here: When developing terraform scripts locally on my laptop, I would prefer to use a local backend. I'll be working in my personal developer azure subscription, and making sure I'm destroying everything after I'm done developing. For our CI pipeline however, I would like to use the http backend to store the state file.

Currently there's no easy way to switch between backends, besides having 2 configurations in the *.tf files and making sure one of them is commented out (or replacing local with http or similar). This poses an obvious risk where a develop forgets to rever the backend type back to http, resulting not in a failed CI run (which would be preferable) but in newly created resources in our production environments.

[whiskeysierra]

but in newly created resources in our production environments.

That one can and should be partially mitigated by not granting so broad permissions to all developers by default.

[AlexGoris-KasparSolutions]

but in newly created resources in our production environments.

That one can and should be partially mitigated by not granting so broad permissions to all developers by default.

That is not the issue, the only account with permissions to the production environments is the CI pipeline account, but if developers forget to revert the local backend configuration, that account will not have any statefile about the existing resources, and created everything from scratch.

[whiskeysierra]

Someone should have peer-reviewed that and approved the pull request. From my experience, that's a low chance.

[AlexGoris-KasparSolutions]

Someone should have peer-reviewed that and approved the pull request. From my experience, that's a low chance.

Yes totally agree. Still it's cumbersome having to remember to switch the backend right before committing.

[whiskeysierra]

Something that I've done before is to generate a temporary backend file on the environment that I'm currently running at. backend.tf.json that I generated using jq. My case was a bit different (same terraform modules, but applied to 3 different environments). Could be an option still.

[iGotTheGift]

Still it's cumbersome having to remember to switch the backend right before committing.

Instead of modifying your actual config have you considered just running terraform init -backend-config=PATH_TO_CONFIG

You could just put a file with your local configuration outside of your Terraform directory and not need to worry about accidentally checking in your actual configuration with changes.

[manoadamro]

why does it not error if you use interpolation in these strings? it just goes along all happy, failing to report the fact it's not going store your state, and break absolutely everything you do from this point forward. It's completely silent.

from the docs:

Only one backend may be specified and the configuration may not contain interpolations. Terraform will validate this.

terraform absolutely does not validate this. (if it does, it doesn't report it's findings)

does this not also go against being able to create multiple clusers at the same time? So this is not possible...

cluster-1
  dev
  prod
  demo

cluster-2
  dev
  prod
  demo

[colans]

I believe we can close this given the solution provided at https://github.com/hashicorp/terraform/pull/20428#issuecomment-470674564.

This issue appears to be a duplicate of https://github.com/hashicorp/terraform/issues/13022.

[apparentlymart]

Agreed that this is a duplicate of https://github.com/hashicorp/terraform/issues/13022. I'm going to close this one to consolidate.

If you've added a :+1: upvote to this one and not to the other one, I suggest transferring your vote to the other issue.

Thanks!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.