tenant_id and subscription_id in azurerm backend configuration not honoured in Azure CLI authentication scenario (but ARM variables are)

[t-l-k]

Terraform Version

Terraform v0.13.3

Terraform Configuration Files

terraform {
  backend "azurerm" {
    tenant_id               = "5e600a2c-..."
    subscription_id         = "33d31cd9-..."
    resource_group_name     = "..."
    storage_account_name    = "..."
    container_name          = "..."
    key                         = "..."
  }
}

Debug Output

> terraform init
2020/09/23 14:26:10 [INFO] Terraform version: 0.13.3
2020/09/23 14:26:10 [INFO] Go runtime version: go1.14.7
2020/09/23 14:26:10 [INFO] CLI args: []string{"/usr/bin/terraform", "init"}
2020/09/23 14:26:10 [DEBUG] Attempting to open CLI config file: /home/t-l-k/.terraformrc
2020/09/23 14:26:10 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/09/23 14:26:10 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/09/23 14:26:10 [DEBUG] ignoring non-existing provider search directory /home/t-l-k/.terraform.d/plugins
2020/09/23 14:26:10 [DEBUG] ignoring non-existing provider search directory /home/t-l-k/.local/share/terraform/plugins
2020/09/23 14:26:10 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2020/09/23 14:26:10 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2020/09/23 14:26:10 [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins
2020/09/23 14:26:10 [INFO] CLI command args: []string{"init"}

Initializing the backend...
2020/09/23 14:26:10 [TRACE] Meta.Backend: built configuration for "azurerm" backend with hash value 2366391659
2020/09/23 14:26:10 [TRACE] Preserving existing state lineage "d9913749-be50-6220-8033-442f4762468f"
2020/09/23 14:26:10 [TRACE] Preserving existing state lineage "d9913749-be50-6220-8033-442f4762468f"
2020/09/23 14:26:10 [TRACE] Meta.Backend: working directory was previously initialized for "azurerm" backend
2020/09/23 14:26:10 [TRACE] Meta.Backend: using already-initialized, unchanged "azurerm" backend configuration
2020/09/23 14:26:10 [DEBUG] Loading Environment "public"
2020/09/23 14:26:10 Testing if Service Principal / Client Certificate is applicable for Authentication..
2020/09/23 14:26:10 Testing if Multi Tenant Service Principal / Client Secret is applicable for Authentication..
2020/09/23 14:26:10 Testing if Service Principal / Client Secret is applicable for Authentication..
2020/09/23 14:26:10 Testing if Managed Service Identity is applicable for Authentication..
2020/09/23 14:26:10 Testing if Obtaining a token from the Azure CLI is applicable for Authentication..
2020/09/23 14:26:10 Using Obtaining a token from the Azure CLI for Authentication
2020/09/23 14:26:11 Getting OAuth config for endpoint https://login.microsoftonline.com/ with  tenant 8304fb80-...

Error: Error obtaining Authorization Token from the Azure CLI: Error parsing json result from the Azure CLI: Error waiting for the Azure CLI: exit status 1

Expected Behavior

It should use the tenant_id and subscription_id as configured in the backend.

The documentation appears to indicate that tenant_id and subscription_id are exclusively use for the following scenarios:

• When authenticating using Managed Service Identity (MSI)
• When authenticating using a Service Principal with a Client Certificate
• When authenticating using a Service Principal with a Client Secret

But honestly, I expect it also to work when using Azure CLI, to authenticate against the tenant and subscription I have explicitly stated. I don't want to authenticate using a SPN, or certificate or etc, I just want to use Azure CLI for authentication, to authenticate as me.

Actual Behavior

It appears to just use what ever ambient Azure CLI account I have logged in.

I have multiple az account contexts, across multiple tenants and subscriptions. It is not always convenient to continuously switch my active one, when the correct account authentication context should be resolved via the explicitly set (and unchanging) tenant_id and subscription_id in the backend configuration.

HOWEVER, I tried also setting ARM_SUBSCRIPTION_ID and ARM_TENANT_ID environment variables. To my surprise, this works! It does use the values from those environment variables when performing Azure CLI authentication!

So the documentation is also a bit inconsistent, because it also implies those variables are also exclusively for use by the 3 scenarios under Expected Behaviour.

I'd rather it just used the settings from the backend configuration.

Steps to Reproduce

1) Have multiple tenants & subscriptions authenticated with az the Azure CLI, plus also a storage account for some remote state etc. 2) az account set with the terraform subscription you intend to use. 3) Set tenant_id and subscription_id in the backend configuration to match the storage account from (1) and the account selected in (2). 4) Write some state into the backend, e.g. terraform apply! 5) az account set to a tenant & subscription which is not the same as configured for the backend. 6) terraform init 7) Kabooom!

[apmarshall]

Looking at this, seems like this might be three issues in one:

1. A documentation issue: docs don’t clearly specify how authentication works when using az cli. Presumably, from reading between the lines of the documentation and your report, they default to using current logged-in creds in az (ie, ignoring the tenant_id and subscription_id fields), but this isn’t spelled out.
2. A request for an enhancement: instead of current (presumed) behavior, use tenant_id and subscription_id to override current az cli creds.
3. A consistency issue: apparently using the ARM_TENANT_ID and ARM_SUBSCRIPTION_ID variables does not behave the same way as setting tenant_id and subscription_id in the config.

Is that a fair summary/disambiguation?

[t-l-k]

@apmarshall Yes, a fair summary indeed.

3. **A consistency issue**: apparently using the `ARM_TENANT_ID` and `ARM_SUBSCRIPTION_ID` variables does not behave the same way as setting `tenant_id` and `subscription_id` in the config.

If those values are set as environment variables then they do take effect. But backend tenant_id and subscription_id are ignored regardless of whether the environment variables are set.

To be honest, if those ARM_* environment variables are set, I would expect them to override anything set in the backend configuration anyway, so that appears to work? Although maybe terraform should write out a warning?

[mxk]

Specifying tenant_id and subscription_id with Azure CLI authentication appears to work now. I'm deploying resources to the default subscription as selected in the CLI, but the state is stored in a separate subscription set via the backend config. Can anyone confirm if this issue is fixed? Version information:

$ az version
{
  "azure-cli": "2.21.0",
  "azure-cli-core": "2.21.0",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {}
}
$ tf version
Terraform v0.14.9
+ provider registry.terraform.io/hashicorp/azurerm v2.54.0

[t-l-k]

@mxk it's still exhibiting the odd behaviour for me. It's attempting to use the subscription from az account show instead of the one annotated in the backend's subscription_id.

az-cli v2.23.0
azurerm v2.58.0
terraform v0.14.9

[apeeters]

It seems to work on azurerm v2.68.0, although I don't feel confident enough yet to confirm.

[GuillaumeDesforges]

Still an issue in v2.86.0

[cccs-mikey-boy]

Still an issue in azurerm v3.32.0

[cveld]

@t-l-k To be honest, if those ARM_* environment variables are set, I would expect them to override anything set in the backend configuration anyway, so that appears to work? Although maybe terraform should write out a warning?

I tend to disagree. Configuration in files should always get preference above environment variables.

[fajfer]

@t-l-k To be honest, if those ARM_* environment variables are set, I would expect them to override anything set in the backend configuration anyway, so that appears to work? Although maybe terraform should write out a warning?

I tend to disagree. Configuration in files should always get preference above environment variables.

It's literally the other way around in other projects, ansible comes to my mind first where CLI arguments override all. If you're that specific to input CLI arguments - I'd be mad if they were overriden by some spaghetti inside of whatever i'm running