terraform_remote_state support for workspace prefix

[matti]

I can not give prefix attribute only the full name like this:

data "terraform_remote_state" "ingress" {
  backend = "remote"
  config = {
    organization = "myorg"
    workspaces = {
      name = "ingress-${terraform.workspace}"
    }
  }

giving prefix errors out with

error loading the remote state: default workspace not supported
You can create a new workspace with the "workspace new" command.

even if the workspace is created&selected

related #21506

[moisedo]

@matti It seems that you are trying to retrieve information from an saved state file, am i correct? Also, what is the backend you are using? (Local, azurerm, s3, consul, etc)?

I can help you with the AzureRM backend, here is what works

1st define your backend block:

• Note: Due to limitations, you cannot use interpolations prior TF execution. therefore you must use hard coded values in this section.

  # AzureRM Backend Configuration:
  
  terraform {
  backend "azurerm" {
   storage_account_name = "<Storage_account_name>"
   container_name       = "<name_of_container>"
   key                  = "<name_of_state_file>"
  }
  }

With this block you must have an environment variable called ARM_ACCESS_KEY in order to pass the access key to store this state file remotely (I don't like to store hard coded secrets in the code). Also, as per your question, this block will allow you save your state file name with the following naming convention:

Sample_State_file:Workspace_Name == Network:Dev

The workspace name is attached as a suffix rather than a prefix in this case.

Once you have those 2 things defined, then let's now use the data provider to retrieve information from it:

# TF Storage Account Configuration
data "azurerm_storage_account" "tf_state" {
  name                = var.tf_sa_name
  resource_group_name = var.tf_sa_rg_name
}

# Remote State Data Configuration:
data "terraform_remote_state" "network" {
  backend = "azurerm"
  workspace = terraform.workspace

  config {
    storage_account_name = data.azurerm_storage_account.tf_state.name
    container_name       = var.tf_container
    key                  = "<name_of_state_file>"
    access_key           = data.azurerm_storage_account.tf_state.primary_access_key
  }
}

As you can see, the first thing i am doing is utilizing the azurerm_storage_account data source with some variables that are known to me so i don't have to hard code any storage account names & resource groups, with this now, i proceed with filling in the config block with the information i need.

I hope this helps.

azurerm #backend #statefile #azure #terraform v0.12

[matti]

I'm using terraform enterprise aka "remote" as stated my config.

[lotjuh]

I have a similar problem with this configuration:

terraform {
  backend "remote" {
    hostname = "app.terraform.io"
    organization = "org"

    workspaces {
      prefix = "org-"
    }
  }
}

data "terraform_remote_state" "common" {
  backend = "remote"
  workspace = "${terraform.workspace}"
  config = {
    hostname = "app.terraform.io"
    organization = "org"

    workspaces = {
      name = "${terraform.workspace}"
    }
  }
}

I get the error

Error: Error loading state error

  on main.tf line 13, in data "terraform_remote_state" "remote":
  13:   backend = "remote"

error loading the remote state: Error creating workspace default: resource not
found

It seems that when using the terraform enterprise remote state to select the workspace, it always sets terraform.workspace to 'default' even though the './.terraform/environment' file has the correct workspace set

[Vishwas1976]

am getting similar error when using workspace and remote only when using the prefix..e.g. workspaces { prefix = "blah-" }

however if i use workspaces = { name = "blah-${terraform.workspace}" } ... it works

[akamalov]

Environment: Terraform v0.12.5

My remote file:

terraform {
  backend "remote" {
    hostname = "terraform-xxxxx.xxxxxxxx.xxxxxx.com"
    organization = "devops2"
    workspaces {
     name = "abc-123"
   }
 }
}

Create a workspace:

$ terraform init

Initializing the backend...

Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.
Error loading state: Error creating workspace abc-123: Invalid Attribute

Terraform version "0.12.5" is not a valid version

$

My terraformrc is up-to-date. Should be alright.

A bit baffled..

[galindro]

Same problem here. terraform_remote_state datasource doesn't seems to support prefix in workspace config from remote backend (terraform cloud). I'm using TF 0.12.6.

data "terraform_remote_state" "transit_gateway" {
  backend = "remote"

  config = {
    hostname     = "app.terraform.io"
    organization = "my-org"
    workspaces = {
      prefix = "my-prefix_"
    }
  }
}

$ terraform workspace list
* eu-central-1

$ terraform plan -var-file=eu-central-1.tfvars 
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.transit_gateway: Refreshing state...

Error: Error loading state error

  on main.tf line 14, in data "terraform_remote_state" "transit_gateway":
  14:   backend = "remote"

error loading the remote state: default workspace not supported
You can create a new workspace with the "workspace new" command.

Releasing state lock. This may take a few moments...

[rberlind]

The ${terraform.workspace} interpolation cannot be used when doing runs on Terraform Cloud (TFC)and Terraform Enterprise (TFE). That is because the concept of "workspace" in TFC is different from the one in open source Terraform. In the latter (sometimes called a "local workspace"), the terraform workspace command can be used to set more than one "workspace" for the configuration and the interpolation ${terraform.workspace} can be used to refer to the current workspace set. However, in TFC, each "workspace" (corresponding to one listed in the UI) only ever has the single OSS workspace "default" which is why the interpolation always gives "default".

This does mean that Terraform configurations that use that interpolation must be modified to not use it for TFC if doing remote runs on TFC or TFE.

Please see https://www.terraform.io/docs/cloud/migrate/workspaces.html for instructions on migrating multiple local workspaces to TFC.

However, Terraform OSS workspaces within a single Terraform OSS configuration CAN be mapped to multiple TFC/TFE workspaces (each of which will have the single "default" OSS workspace).

Specifically, if you have a Terraform OSS configuration in a directory with multiple workspaces such as "dev", "qa", and "prod", you can set up the remote backend with the workspaces.prefix option set to something like my-workspaces-. If you run terraform workspace select dev and then run terraform apply, the run on the Terraform Cloud (or Terraform Enterprise) server will be done in a workspace called "my-workspaces-dev". If you run terraform workspace select qa and then run terraform apply, the run on the Terraform Cloud (or Terraform Enterprise) server will be done in a workspace called "my-workspaces-qa". If you run terraform workspace select prod and then run terraform apply, the run on the Terraform Cloud (or Terraform Enterprise) server will be done in a workspace called "my-workspaces-prod".

Note that the ${terraform.workspace} interpolation is NOT used when doing this mapping. Instead, the Terraform OSS workspace name ("dev", "qa", or "prod") will automatically be used in the name of the TFC/TFE workspace when doing runs on the remote server. (The same applies with regard to storing of state in TFC/TFE workspaces if doing local runs.)

The bottom line with regard to the ${terraform.workspace} interpolation is that it should not be used in Terraform configurations that are doing remote runs on TFC or TFE. It should be ok to use it if you are only doing local runs and only using TFC/TFE to store remote state.

Note that the TFC/TFE workspaces will be created for you if they do not already exist and will use the most recent version of Terraform OSS. However, if your workspace needs variables set, especially environment variables such as cloud credentials, you should manually create the workspaces and set any needed variables. If you want to use a version of Terraform OSS for the runs done on the TFC/TFE server in those workspaces, then you should also set the version before your first plan or apply.

Regarding use of remote backend with the terraform_remote_state data source, the workspace containing the state that is specified in that data source's config.workspaces stanza can only specify a name attribute and that using prefix would not be allowed. See the terraform_remote_state documentation: https://www.terraform.io/docs/providers/terraform/d/remote_state.html as well as the actual example under the remote backend doc: https://www.terraform.io/docs/backends/types/remote.html#example-reference

Hopefully, that clarifies how Terraform OSS workspaces can be mapped to TFC/TFE workspaces with the remote backend configuration as well as why the ${terraform.workspace} interpolation should not be used in Terraform configurations doing remote runs on TFC/TFE servers.

[Sytten]

@rberlind Its all good to say you cant use the ${terraform.workspace} interpolation, but what should we use instead? It is a very common use case to have differences between prod and other environments and it is also common to happen the environment name to the resources names.

[rberlind]

Unfortunately, @Sytten , I don't have a good answer for you. I did file an enhancement request with the Terraform product management team. But I have no control over the timing of any changes they might or might not make to handle that request.

[fewknow]

This is needed ASAP.

[timorthi]

Same issue here with Terraform Cloud.

I've resorted to using an input variable as the name in the data source:

data "terraform_remote_state" "vpc" {
  backend = "remote"

  config = {
    organization = "MyOrg"

    workspaces = {
      name = var.networking_remote_state_workspace_name
    }
  }
}

It gets the job done, though I would much rather just do:

...
workspaces = {
  prefix = "vpc-"
}
...

[matti]

updated the issue title: this is very valid in 2021 also