hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.51k stars 4.6k forks source link

Deployment of a subscription via MPA terminates with InvalidAuthenticationTokenTenant #22260

Open oed-guzym opened 1 year ago

oed-guzym commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.5.1

AzureRM Provider Version

3.61.0

Affected Resource(s)/Data Source(s)

azurerm_subscription

Terraform Configuration Files

# versions.tf
terraform {
  required_version = ">= 1.4"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3"
    }
  }
}

# providers.tf
provider "azurerm" {
  features {}

  tenant_id = "<CSP_TENANT_ID>"
}

# main.tf
data "azurerm_billing_mpa_account_scope" "this" {
  billing_account_name = "<CSP_BILLING_ID>"
  customer_name        = "<CUSTOMER_TENANT_ID>"
}

resource "azurerm_subscription" "this" {
  subscription_name = "SUBSCRIPTION-TST-EXAMPLE-001"
  billing_scope_id  = data.azurerm_billing_mpa_account_scope.this.id
  workload          = "Production"
}

Debug Output/Panic Output

{
  "error": { 
    "code": "InvalidAuthenticationTokenTenant",
    "message": "The access token is from the wrong issuer 'https://sts.windows.net/<CSP_TENANT_ID>/'. It must match the tenant 'https://sts.windows.net/<CUSTOMER_TENANT_ID>/' associated with this subscription. Please use the authority (URL) 'https://login.windows.net/<CUSTOMER_TENANT_ID>' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later."
  }
}

2023-06-23T10:10:37.466+0200 [ERROR] provider.terraform-provider-azurerm_v3.61.0_x5: Response contains error diagnostic: diagnostic_summary="failed waiting for Subscription "<SUBSCRIPTION_ID>" (Alias "<ALIAS>") to enter "Active" state: waiting for Subscription "<SUBSCRIPTION_ID>" to become "Active", currently """ tf_proto_version=5.3 tf_provider_addr=provider tf_req_id=510cd2ef-fc71-de0d-27de-bec2a9a8777e tf_resource_type=azurerm_subscription @module=sdk.proto diagnostic_detail= diagnostic_severity=ERROR tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 timestamp=2023-06-23T10:10:37.466+0200

2023-06-23T10:10:37.489+0200 [ERROR] vertex "azurerm_subscription.this" error: failed waiting for Subscription "<SUBSCRIPTION_ID>" (Alias "<ALIAS>") to enter "Active" state: waiting for Subscription "<SUBSCRIPTION_ID>" to become "Active", currently ""
╷
│ Error: failed waiting for Subscription "<SUBSCRIPTION_ID>" (Alias "<ALIAS>") to enter "Active" state: waiting for Subscription "<SUBSCRIPTION_ID>" to become "Active", currently ""
│
│   with azurerm_subscription.this,
│   on main.tf line 8, in resource "azurerm_subscription" "this":
│    8: resource "azurerm_subscription" "this" {

Expected Behaviour

Our expectation is that we can create a subscription through Terraform for a customer tenant via the Microsoft Partner Agreement through our CSP tenant.

Actual Behaviour

Currently, we manually create a new subscription for a customer tenant from the Azure portal via our CSP tenant. We would like to automate this step using the "azurerm_subscription" resource.

In the provider configuration, the tenant ID of our CSP tenant is configured and the Terraform code itself is executed via a personalized user that exists in the Active Directory of the CSP tenant as Global Admin.

After running the code, we get the mentioned error message saying that the wrong token seems to be used and Terraform terminates the deployment. Interestingly, the subscription is still created in the customer tenant, but without tags.

In the documentation of the Terraform resource you can read the following:

When using this resource across tenants the client_id and tenant_id of the provider config block should be for the home tenant details for the SPN / User or a permissions error will likely be encountered. See the official documentation for more details

If we look at the official documentation for Programmatically create Azure subscriptions for a Microsoft Partner Agreement with the latest APIs, it says the following as a prerequisite:

You must have a Global Admin or Admin Agent role in your organization's cloud solution provider account to create subscription for your billing account. For more information, see Partner Center - Assign users roles and permissions.

Unfortunately, I have no idea what the configuration via Terraform should look like so that we can automatically provision a subscription in a customer tenant. The fact that a personalized user was used is just a test. In the end, a service principal should be used for provisioning.

Does anyone have any idea what the provider configuration should look like or exactly what pre-requisites for Terraform need to be met in order for the deployment to go through without error message?

Steps to Reproduce

No response

Important Factoids

No response

References

No response

oed-guzym commented 1 year ago

Hi all, I tried the steps described in the documentation to Programmatically create Azure subscriptions for a Microsoft Partner Agreement with the latest APIs using the Azure CLI and my CSP user and was able to create a subscription in a custom tenant without any issues.

### Request
###########

billingAccount=$( \
  az billing account list | jq -r '.[] | select(.displayName | contains("<CSP Tenant>")) | .name' \
)

customerTenantId=$( \
  az billing customer list \ 
    --account-name ${billingAccount} \
    | jq -r '.[] | select(.displayName | contains("<Customer>")) | .name' \
)

az account alias create \ 
  --name "SUBSCRIPTION-TST-EXAMPLE-CLI" \ 
  --billing-scope "/providers/Microsoft.Billing/billingAccounts/${billingAccount}/customers/${customerTenantId}" \
  --display-name "SUBSCRIPTION-TST-EXAMPLE-CLI" \
  --workload "Production"

### Response
############

{
  "id": "/providers/Microsoft.Subscription/aliases/SUBSCRIPTION-TST-EXAMPLE-CLI",
  "name": "SUBSCRIPTION-TST-EXAMPLE-CLI",
  "properties": {
    "acceptOwnershipState": null,
    "acceptOwnershipUrl": null,
    "billingScope": null,
    "createdTime": null,
    "displayName": null,
    "managementGroupId": null,
    "provisioningState": "Succeeded",
    "resellerId": null,
    "subscriptionId": "<SUBSCRIPTION_ID>",
    "subscriptionOwnerId": null,
    "tags": null,
    "workload": null
  },
  "systemData": null,
  "type": "Microsoft.Subscription/aliases"
}

In theory, the steps should not differ too much via Terraform, because there we also link the Billing ID of the CSP tenant and the respective tenant ID of the customer. I also run the Terraform code through my CSP user, so lack of permissions shouldn't really be a problem.

Since we mainly work with Terraform, we would also like to realize the creation of a subscription via Terraform. The Azure CLI would be more a workaround and not a real alternative. However, I don't know what our problem is with Terraform and how we can solve it.

oed-guzym commented 1 year ago

Are there any updates on this issue?

giridhar-kolli commented 7 months ago

Hello All, We also came across this same issue recently and are actively looking for solutions. We used azapi_resource to create a subscription, also using a Service Principal to provision and have added in provider.

Terraform Configuration File

 # provider.tf
terraform {
  required_version = ">= 1.3.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.7.0"
    }
    azapi = {
      source  = "Azure/azapi"
      version = ">= 1.3.0"
    }
  }
}

provider "azurerm" {
  features {}
  client_id = "<SP_Client_ID>"
  client_secret = "<SP_Secret_Value>"
  skip_provider_registration = true
  tenant_id = var.CSP_tenant_id
}

 # main.tf
data "azurerm_billing_mpa_account_scope" "CSP_Tenant" {
  billing_account_name = "CSP_Tenant_Billing_account_Name"
  customer_name = var.customer_tenant_id

}
resource "azapi_resource" "subscription" {
  type      = "Microsoft.Subscription/aliases@2021-10-01"
  name      = var.subscription_alias_name 
  parent_id = "/"

  body = jsonencode({
    properties = {
      displayName  = var.subscription_name
      workload     = "Production"
      billingScope = data.azurerm_billing_mpa_account_scope.CSP_Tenant.id 
    }
  })
  response_export_values = ["properties.subscriptionId"]
  lifecycle {
    ignore_changes = [
      body,
      name
    ]
  }
}

Output

2024-01-26T23:09:13.274Z [ERROR] provider.terraform-provider-azurerm_v3.89.0_x5: Response contains error diagnostic: @module=sdk.proto tf_provider_addr=provider @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="building account: unable to configure ResourceManagerAccount: subscription ID could not be determined and was not specified" tf_proto_version=5.4 tf_req_id=dab60aff-1fc3-3994-f349-1f61616df1ea tf_rpc=Configure timestamp=2024-01-26T23:09:13.274Z
2024-01-26T23:09:13.275Z [ERROR] vertex "module.sub_vending.provider[\"registry.terraform.io/hashicorp/azurerm\"]" error: building account: unable to configure ResourceManagerAccount: subscription ID could not be determined and was not specified
╷
│ Error: building account: unable to configure ResourceManagerAccount: subscription ID could not be determined and was not specified
│ 
│   with module.sub_vending.provider["registry.terraform.io/hashicorp/azurerm"],
│   on ../../provider.tf line 15, in provider "azurerm":
│   15: provider "azurerm" {
│ 
╵

Unfortunately, It is looking for a subscription_id, which at this point is unknown and we cannot set on the provider. How to proceed on this?

References:

iTiamo commented 1 month ago

I am running into the same issue, any updates?

What I am seeing in the debug logs is that Terraform is waiting for the deployed subscription to be enabled. To do that it calls /subscriptions/<> every minute or so. But since the subscription is in a different tenant the token for our tenant is not valid.