Unable to set block values in `terraform init -backend-config=...`

[lawliet89]

Terraform Version

Terraform v0.12.2
(independent of Terraform Providers)

Terraform Configuration Files

terraform {
  required_version = ">= 0.12"

  backend "remote" {
  }
}

# The rest are irrelevant

Expected Behavior

Consider the usage of the Terraform remote backend where configuration values are provided via the CLI.

Previously, in Terraform 0.11, I could do

terraform init -backend-config=organization=my-org \
  -backend-config=workspaces=[{name = "foobar"}]

I was able to set the values for the workspaces block for the remote backend via -backend-config CLI flag during terraform init.

Actual Behavior

Initializing the backend...

Error: Invalid backend configuration argument

The backend configuration argument "workspaces" given on the command line is
not expected for the selected backend type.

Steps to Reproduce

terraform init -backend-config=organization=my-org \
  -backend-config=workspaces=[{name = "foobar"}]

Additional Context

This change in behaviour is probably due to the commit which added validation against the Attributes of backend schemas only and does not bother with BlockTypes. As far as I could tell, there was no validation previously and the value was passed to the backend directly for deserialization.

This will affect https://github.com/gruntwork-io/terragrunt/issues/687.

[apparentlymart]

Thanks for reporting this, @lawliet89.

Indeed, before this worked as an accidental consequence of how Terraform was using HCL, and no longer works because workspace isn't a list of objects but rather a nested block.

I think we'll need to investigate a little deeper to see what is feasible as a response here. We might be able to address this by having the -backend-config handling check if the given name is of a block and specifically ask HCL to parse the argument as a list of objects in that case before converting it into the correct block representation. This means that we'd be bypassing some of the usual processing HCL does and so we'd likely have to implement some local validation for the contents of the objects in the list, but that should just be annoying rather than impossible.

In the meantime, passing this option via a file on disk would be the workaround, since an HCL file for backend config can use all of the HCL syntax features natively. However, I expect that generating a temporary file like that will not be convenient for terragrunt; generating a file on disk happens to how Terraform Cloud and Terraform Enterprise handle this situation, but they have more control over the execution environment than terragrunt tends to, given that terragrunt can be used directly as a CLI tool.

[jfpanisset]

Ran into this issue as well, I was able to use a disk file to specify the workspace, but spent time trying to get the workspaces=[{name=foo}] syntax working before I found this GitHub Issue. Until this can be resolved, would it make sense to add a note to the documentation at:

https://www.terraform.io/docs/backends/config.html#partial-configuration

to indicate that not being able to set the workspace name/prefix from the command line using -backend-config is a known limitation?

[pjaintaylor]

+1 for this, will definitely help with generating repository-based dynamic workspaces with Terraform Cloud!

[osulli]

Also encountered this for Terratest.

A documentation update would have been useful. Like @jfpanisset I also spent some hours trying to work this out. I'll likely go down the route of specifying an HCL file.

[edmd]

We need to be able to pass variables on terraform init. This is critical.

We can't specify a file with sensitive information.

[chenjianjx]

This is critical. The testing and production should be using different workspaces, so their names has to be dynamic

[tovbinm]

Same here. Any ETA on a fix / workaround?

[osulli]

@chenjianjx @tovbinm Yes, a workaround is to have the workspace in a file and then at runtime move it. Naturally, this would not work in TFC, but would in CI tool where you can run a cp config/blah_backend.tf backend.tf

[brettjacobson]

This really needs to be fixed!

[VladyslavKurmaz]

I spent a day trying to figure out proper solution, until I found this thread. At least I'm not alone with it. Issue is still here in terraform-1.2.6. Backend configuration is one of the most painful part of Terraform. It will be great if we can see fix in one of the upcoming releases.

[apparentlymart]

Hi all! Sorry for the long silence here.

I think this issue got a bit neglected because the "remote" backend has subsequently been replaced by the native Terraform Cloud settings, which achieve a similar result but using a first-class cloud block and some corresponding environment variables.

For the configurations shown in the initial issue report, the equivalent settings for the Terraform Cloud integration would be as follows:

terraform {
  required_version = ">= 1.1.0"

  cloud {
    # (normally settings would go here, but I'm going to show
    # how to set them via environment variables instead below)
  }
}

$ export TF_CLOUD_ORGANIZATION='my-org'
$ export TF_WORKSPACE='foobar'
$ terraform init

There's some more detailed migration information in Migrating from the remote Backend in the Terraform Cloud documentation.

There's more information on the environment variables that can substitute for the settings in the cloud block in Environment Variables.

The remote backend will be preserved for the foreseeable future to allow upgrading Terraform CLI without forcing a migration to the native integration, but no further development is planned for that backend and so its existing limitation of requiring a separate file to configure the workspaces block will remain. If you need to set the Cloud integration settings outside of your configuration then you should migrate to the native Terraform Cloud integration instead, and use the documented environment variables.

I've done my best to give a general overview of the new capabilities here, but if you have a situation that I haven't considered in this comment please contact HashiCorp Support for help with your specific situation.

Thanks for reporting this, and sorry again for the long silence.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.