Terraform Import does not work with Terraform Cloud Varsets

[zackdotcomputer]

TLDR; The variable set feature for Terraform Cloud has not been properly integrated with Terraform CLI local operations like terraform import, meaning that this command fails saying an input "is not set, and has no default value" if that input is required for your application and also would be provided by the remote variable set.

Terraform Version

Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/auth0/auth0 v1.0.0

Terraform Configuration Files

The versions.tf file:

terraform {
  required_version = ">= 1.4.6"

  cloud {
    organization = "<ORG_NAME>"
    workspaces {
      tags = ["auth0"]
    }
  }
}

(note that in the actual project it's not ORG_NAME but rather our actual org name)

The variables.tf file:

variable "AUTH0_DOMAIN" {}
variable "AUTH0_CLIENT_ID" {}
variable "AUTH0_CLIENT_SECRET" {}
variable "API_DOMAIN" {}
variable "FRONTEND_DOMAIN" {}

The main.tf is not relevant other than that it declares resources like an auth0_client

Debug Output

The relevant line is right before the failure:

2023-09-27T11:29:45.856+0100 [TRACE] cloud: retrieving variables from workspace staging-auth0/<ORG_NAME> (ws-<REDACTED>)

(I've redacted the values in the <> because they might expose our settings more precisely.)

Expected Behavior

Running terraform import auth0_client.frontend <CLIENT_ID> should import that client.

Actual Behavior

The process fails with the error:

Warning: Value for var.AUTH0_CLIENT_SECRET unavailable
│ 
│ The value of variable "AUTH0_CLIENT_SECRET" is marked as sensitive in the remote workspace. This operation always runs locally, so the value for that variable is not
│ available.
╵

╷
│ Error: No value for required variable
│ 
│   on /Users/zack/code/infrastructure/applications/auth0/variables.tf line 25:
│   25: variable "API_DOMAIN" {}
│ 
│ The root module input variable "API_DOMAIN" is not set, and has no default value. Use a -var or -var-file command line argument to provide a value for this variable.
╵

╷
│ Error: No value for required variable
│ 
│   on /Users/zack/code/infrastructure/applications/auth0/variables.tf line 27:
│   27: variable "FRONTEND_DOMAIN" {}
│ 
│ The root module input variable "FRONTEND_DOMAIN" is not set, and has no default value. Use a -var or -var-file command line argument to provide a value for this variable.
╵

Steps to Reproduce

1. Create your terraform workspace in terraform cloud.
2. Add it to a Terraform Cloud Project - see https://www.hashicorp.com/blog/terraform-cloud-adds-projects-to-organize-workspaces-at-scale
3. Provide some variables via the variable set feature, and some variables via the workspace's own variables page.
4. Run a terraform import ... command. It should fail.

The fact that the import command is unable to download private variables is already documented in #26494, so that is not the new issue here. The new issue here is that the import command cannot access non-sensitive variables provided by variable sets.

Additional Context

Note that the terraform plan command is able to access all variables and so it does not fail. Additionally, note that the terraform import command only throws an error for missing variables for those provided by the variable set, so it is able to access variables provided at the workspace level.

This is how I narrowed the issue down to be that terraform import specifically cannot access variables defined by the variable sets feature in terraform cloud.

References

No response

[radditude]

Thanks for reporting! Further investigation indicates that this problem is not specific to project varsets, but is a gap in support for variable sets as a whole when looking up variables for local operations - that is, the cloud integration only looks for non-sensitive variables defined locally within the workspace and ignores variable sets entirely, whether or not they're scoped to a specific project. We'll triage this with the appropriate Terraform Cloud team.

In the meantime, you may find the import block that was added in Terraform 1.5 useful as a way to move forward. Unlike the standalone import command, import blocks run remotely in TFC as part of the plan and apply workflow, so they have full access to all credentials and variables, including those provided as variable sets.

[Farbaks]

Hello, seems like the id passed into this import block has to be a string, and therefore the id cannot be passed as a sensitive variable...is there any workaround for this?

[holograph]

As @radditude said, an import block is a possible workaround, but that comes with two problems: 1. it requires (for source controlled workspaces) a full commit-push-approve cycle, and 2. it runs afoul of the occasionally required operator intervention.

In my particular case an AWS runtime error left a resource in a "corrupt" state and I want to remove it from state and reimport to restore functionality; however there's no way to do this without a commit (and corresponding development/review/build cycles).