Open nerddtvg opened 6 days ago
Thanks for raising this issue. Seems I can't reproduce with below steps. Could you try below tfconfig to see if the issue still exists? If my steps is not expected, please correct me. Thanks.
First apply with tf config:
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "test" {
name = "acctestRG-cosmos-test03"
location = "eastus"
}
resource "azurerm_cosmosdb_account" "test" {
name = "acctest-ca-test03"
location = azurerm_resource_group.test.location
resource_group_name = azurerm_resource_group.test.name
offer_type = "Standard"
kind = "GlobalDocumentDB"
default_identity_type = "FirstPartyIdentity"
consistency_policy {
consistency_level = "Eventual"
}
geo_location {
location = azurerm_resource_group.test.location
failover_priority = 0
}
}
Second apply with tf config:
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "test" {
name = "acctestRG-cosmos-test03"
location = "eastus"
}
resource "azurerm_user_assigned_identity" "test" {
resource_group_name = azurerm_resource_group.test.name
location = azurerm_resource_group.test.location
name = "acctest-user-test03"
}
resource "azurerm_cosmosdb_account" "test" {
name = "acctest-ca-test03"
location = azurerm_resource_group.test.location
resource_group_name = azurerm_resource_group.test.name
offer_type = "Standard"
kind = "GlobalDocumentDB"
default_identity_type = join("=", ["UserAssignedIdentity", azurerm_user_assigned_identity.test.id])
identity {
type = "UserAssigned"
identity_ids = [azurerm_user_assigned_identity.test.id]
}
consistency_policy {
consistency_level = "Eventual"
}
geo_location {
location = azurerm_resource_group.test.location
failover_priority = 0
}
}
I'm also not able to reproduce it from this code (modified to include CMK). I'm not sure if re-running it on our existing infrastructure is going to be possible, but I can provide some debug logging form last week to show this isn't quite a phantom issue.
I think the root cause is a bad import or apply putting a bad default_identity_type
value into the state. I tried creating the Cosmos account from the "Part 1" code, removing from the state, importing, and applying "Part 2" but it also succeeded. However, looking at last week's apply, the plan detected a drift where default_identity_type
was UMI, drift was the value from Azure API was FirstPartyIdentity
, and then the plan re-applied the UMI. So I think the apply is using the state value of default_identity_type
rather than the detected drift value.
The apply last week was using an older version of the module requiring 3.117.0.
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~>4.0"
}
}
}
provider "azurerm" {
# Configuration options
features {}
subscription_id = ""
}
variable "cosmos_name" {
type = string
nullable = false
default = "acctest-ca-test03a"
}
variable "tenant_id" {
type = string
nullable = false
default = ""
}
variable "resource_group_name" {
type = string
nullable = false
default = ""
}
variable "key_vault_rg" {
type = string
nullable = false
default = ""
}
variable "key_vault_name" {
type = string
nullable = false
default = ""
}
data "azurerm_resource_group" "test" {
name = var.resource_group_name
}
data "azurerm_key_vault" "test" {
name = var.key_vault_name
resource_group_name = var.key_vault_rg
}
resource "azurerm_key_vault_key" "test" {
name = var.cosmos_name
key_vault_id = data.azurerm_key_vault.test.id
key_type = "RSA"
key_size = 3072
curve = null
key_opts = [
"encrypt",
"decrypt",
"sign",
"verify",
"wrapKey",
"unwrapKey"
]
}
resource "azurerm_key_vault_access_policy" "test_fpi" {
key_vault_id = data.azurerm_key_vault.test.id
object_id = "1d4aa0cb-2b2c-4b19-8117-a64262410b32" # Azure CosmosDB First Party Identity
tenant_id = var.tenant_id
key_permissions = ["Backup", "Create", "Decrypt", "Delete", "Encrypt", "Get", "Import", "List", "Purge", "Recover", "Restore", "Sign", "UnwrapKey", "Update", "Verify", "WrapKey", "Release", "Rotate", "GetRotationPolicy", "SetRotationPolicy"]
}
resource "azurerm_cosmosdb_account" "test" {
name = var.cosmos_name
location = data.azurerm_resource_group.test.location
resource_group_name = data.azurerm_resource_group.test.name
offer_type = "Standard"
kind = "GlobalDocumentDB"
key_vault_key_id = azurerm_key_vault_key.test.versionless_id
# Part 1
# default_identity_type = "FirstPartyIdentity"
# Part 2
default_identity_type = join("=", ["UserAssignedIdentity", azurerm_user_assigned_identity.test.id])
identity {
type = "UserAssigned"
identity_ids = [azurerm_user_assigned_identity.test.id]
}
consistency_policy {
consistency_level = "Eventual"
}
geo_location {
location = data.azurerm_resource_group.test.location
failover_priority = 0
}
depends_on = [
azurerm_key_vault_access_policy.test_fpi,
azurerm_key_vault_access_policy.test_umi
]
}
# Part 2
resource "azurerm_user_assigned_identity" "test" {
resource_group_name = data.azurerm_resource_group.test.name
location = data.azurerm_resource_group.test.location
name = "acctest-user-test03"
}
resource "azurerm_key_vault_access_policy" "test_umi" {
key_vault_id = data.azurerm_key_vault.test.id
object_id = azurerm_user_assigned_identity.test.principal_id
tenant_id = var.tenant_id
key_permissions = ["Backup", "Create", "Decrypt", "Delete", "Encrypt", "Get", "Import", "List", "Purge", "Recover", "Restore", "Sign", "UnwrapKey", "Update", "Verify", "WrapKey", "Release", "Rotate", "GetRotationPolicy", "SetRotationPolicy"]
}
Plan Output:
2024-11-22T21:37:29.362Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: GET https://management.azure.com/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15: timestamp=2024-11-22T21:37:29.362Z
2024-11-22T21:37:29.500Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15:
HTTP/2.0 200 OK
Content-Length: 3737
Cache-Control: no-store, no-cache
Content-Type: application/json
Date: Fri, 22 Nov 2024 21:37:28 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: a90f95c0-b735-1db8-e3a4-fefd8ce4c33e
X-Ms-Gatewayversion: version=2.14.0
X-Ms-Ratelimit-Remaining-Subscription-Global-Reads: 3749
X-Ms-Ratelimit-Remaining-Subscription-Reads: 249
X-Ms-Request-Id: 3672fa5a-3303-4d62-a1bf-239e0e858f16
X-Ms-Routing-Request-Id: WESTUS:20241122T213729Z:3672fa5a-3303-4d62-a1bf-239e0e858f16
X-Msedge-Ref: Ref A: 459C4A220C6C4FBDBCB9293C9EABA98F Ref B: SJC211051204051 Ref C: 2024-11-22T21:37:29Z
{"id":"/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name","name":"cosmos_name","location":"West US 2","type":"Microsoft.DocumentDB/databaseAccounts","kind":"GlobalDocumentDB","tags":{},"systemData":{"createdAt":"2023-03-31T17:36:07.3308042+00:00"},"properties":{"provisioningState":"Succeeded","documentEndpoint":"https://cosmos_name.documents.azure.com:443/","sqlEndpoint":"https://cosmos_name.documents.azure.com:443/","publicNetworkAccess":"Disabled","enableAutomaticFailover":false,"enableMultipleWriteLocations":false,"enablePartitionKeyMonitor":true,"isVirtualNetworkFilterEnabled":false,"virtualNetworkRules":[],"privateEndpointConnections":[{"id":"/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name/privateEndpointConnections/cosmos_name-pe-Sql","properties":{"privateEndpoint":{"id":"/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.Network/privateEndpoints/cosmos_name-pe-Sql"},"privateLinkServiceConnectionState":{"status":"Approved","actionsRequired":"None"}}}],"EnabledApiTypes":"Sql","disableKeyBasedMetadataWriteAccess":false,"enableFreeTier":false,"enableAnalyticalStorage":false,"analyticalStorageConfiguration":{"schemaType":"WellDefined"},"instanceId":"9b6d68bb-a35b-4517-958e-37a0e6e00863","databaseAccountOfferType":"Standard","defaultIdentity":"FirstPartyIdentity","networkAclBypass":"None","disableLocalAuth":false,"enablePartitionMerge":false,"enableBurstCapacity":false,"minimalTlsVersion":"Tls12","keyVaultKeyUri":"https://key_vault_name.vault.azure.net/keys/cosmos_name","customerManagedKeyStatus":"Access to the configured customer managed key confirmed. ","consistencyPolicy":{"defaultConsistencyLevel":"Eventual","maxIntervalInSeconds":5,"maxStalenessPrefix":100},"configurationOverrides":{},"writeLocations":[{"id":"cosmos_name-westus2","locationName":"West US 2","documentEndpoint":"https://cosmos_name-westus2.documents.azure.com:443/","provisioningState":"Succeeded","failoverPriority":0,"isZoneRedundant":true}],"readLocations":[{"id":"cosmos_name-westus2","locationName":"West US 2","documentEndpoint":"https://cosmos_name-westus2.documents.azure.com:443/","provisioningState":"Succeeded","failoverPriority":0,"isZoneRedundant":true}],"locations":[{"id":"cosmos_name-westus2","locationName":"West US 2","documentEndpoint":"https://cosmos_name-westus2.documents.azure.com:443/","provisioningState":"Succeeded","failoverPriority":0,"isZoneRedundant":true}],"failoverPolicies":[{"id":"cosmos_name-westus2","locationName":"West US 2","failoverPriority":0}],"cors":[],"capabilities":[],"ipRules":[],"backupPolicy":{"type":"Periodic","periodicModeProperties":{"backupIntervalInMinutes":240,"backupRetentionIntervalInHours":720,"backupStorageRedundancy":"Geo"}},"networkAclBypassResourceIds":[],"keysMetadata":{"primaryMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"},"secondaryMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"},"primaryReadonlyMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"},"secondaryReadonlyMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"}}},"identity":{"type":"None"}}
: timestamp=2024-11-22T21:37:29.500Z
2024-11-22T21:37:30.205Z [WARN] Provider "registry.terraform.io/hashicorp/azurerm" produced an unexpected new value for module.azure_baseline.module.cosmosdb["key"].azurerm_cosmosdb_account.cosmosdb_account during refresh.
- .default_identity_type: was cty.StringVal("UserAssignedIdentity=/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/umi_name"), but now cty.StringVal("FirstPartyIdentity")
Apply log:
2024-11-22T21:38:30.417Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: GET https://management.azure.com/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15: timestamp=2024-11-22T21:38:30.417Z
2024-11-22T21:38:30.559Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15:
HTTP/2.0 200 OK
Content-Length: 3737
Cache-Control: no-store, no-cache
Content-Type: application/json
Date: Fri, 22 Nov 2024 21:38:29 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: c74512b4-e300-c630-5586-b649468bf01d
X-Ms-Gatewayversion: version=2.14.0
X-Ms-Ratelimit-Remaining-Subscription-Global-Reads: 3749
X-Ms-Ratelimit-Remaining-Subscription-Reads: 249
X-Ms-Request-Id: ad5db01c-4d27-4602-80b9-09f00c1a510b
X-Ms-Routing-Request-Id: WESTUS:20241122T213830Z:ad5db01c-4d27-4602-80b9-09f00c1a510b
X-Msedge-Ref: Ref A: 3F6E0C3AB021472581F9D6994D8FAB05 Ref B: SJC211051203049 Ref C: 2024-11-22T21:38:30Z
{"id":"/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name","name":"cosmos_name","location":"West US 2","type":"Microsoft.DocumentDB/databaseAccounts","kind":"GlobalDocumentDB","tags":{"AzShPr":"U2FsdGVkX1/HbyXvkyxMjBCz5baeXA3H8R8pIUw0jetwCmvDC0R9ASIXBojbOfex"},"systemData":{"createdAt":"2023-03-31T17:36:07.3308042+00:00"},"properties":{"provisioningState":"Succeeded","documentEndpoint":"https://cosmos_name.documents.azure.com:443/","sqlEndpoint":"https://cosmos_name.documents.azure.com:443/","publicNetworkAccess":"Disabled","enableAutomaticFailover":false,"enableMultipleWriteLocations":false,"enablePartitionKeyMonitor":true,"isVirtualNetworkFilterEnabled":false,"virtualNetworkRules":[],"privateEndpointConnections":[{"id":"/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name/privateEndpointConnections/cosmos_name-pe-Sql","properties":{"privateEndpoint":{"id":"/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.Network/privateEndpoints/cosmos_name-pe-Sql"},"privateLinkServiceConnectionState":{"status":"Approved","actionsRequired":"None"}}}],"EnabledApiTypes":"Sql","disableKeyBasedMetadataWriteAccess":false,"enableFreeTier":false,"enableAnalyticalStorage":false,"analyticalStorageConfiguration":{"schemaType":"WellDefined"},"instanceId":"9b6d68bb-a35b-4517-958e-37a0e6e00863","databaseAccountOfferType":"Standard","defaultIdentity":"FirstPartyIdentity","networkAclBypass":"None","disableLocalAuth":false,"enablePartitionMerge":false,"enableBurstCapacity":false,"minimalTlsVersion":"Tls12","keyVaultKeyUri":"https://key_vault_name.vault.azure.net/keys/cosmos_name","customerManagedKeyStatus":"Access to the configured customer managed key confirmed. ","consistencyPolicy":{"defaultConsistencyLevel":"Eventual","maxIntervalInSeconds":5,"maxStalenessPrefix":100},"configurationOverrides":{},"writeLocations":[{"id":"cosmos_name-westus2","locationName":"West US 2","documentEndpoint":"https://cosmos_name-westus2.documents.azure.com:443/","provisioningState":"Succeeded","failoverPriority":0,"isZoneRedundant":true}],"readLocations":[{"id":"cosmos_name-westus2","locationName":"West US 2","documentEndpoint":"https://cosmos_name-westus2.documents.azure.com:443/","provisioningState":"Succeeded","failoverPriority":0,"isZoneRedundant":true}],"locations":[{"id":"cosmos_name-westus2","locationName":"West US 2","documentEndpoint":"https://cosmos_name-westus2.documents.azure.com:443/","provisioningState":"Succeeded","failoverPriority":0,"isZoneRedundant":true}],"failoverPolicies":[{"id":"cosmos_name-westus2","locationName":"West US 2","failoverPriority":0}],"cors":[],"capabilities":[],"ipRules":[],"backupPolicy":{"type":"Periodic","periodicModeProperties":{"backupIntervalInMinutes":240,"backupRetentionIntervalInHours":720,"backupStorageRedundancy":"Geo"}},"networkAclBypassResourceIds":[],"keysMetadata":{"primaryMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"},"secondaryMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"},"primaryReadonlyMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"},"secondaryReadonlyMasterKey":{"generationTime":"2023-03-31T17:36:07.3308042+00:00"}}},"identity":{"type":"None"}}
: timestamp=2024-11-22T21:38:30.559Z
2024-11-22T21:38:30.561Z [INFO] provider.terraform-provider-azurerm_v3.117.0_x5: Updating AzureRM Cosmos DB Account: Updating 'DatabaseAccountCreateUpdateParameters': timestamp=2024-11-22T21:38:30.561Z
2024-11-22T21:38:30.561Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: AzureRM Request:
PUT /subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 cosmosdb/2024-05-15) HashiCorp Terraform/1.7.5 (+https://www.terraform.io) terraform-provider-azurerm/3.117.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 1547
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: c74512b4-e300-c630-5586-b649468bf01d
Accept-Encoding: gzip
{"kind":"GlobalDocumentDB","location":"westus2","properties":{"analyticalStorageConfiguration":{"schemaType":"WellDefined"},"backupPolicy":{"periodicModeProperties":{"backupIntervalInMinutes":240,"backupRetentionIntervalInHours":720,"backupStorageRedundancy":"Geo"},"type":"Periodic"},"capabilities":[],"consistencyPolicy":{"defaultConsistencyLevel":"Eventual","maxIntervalInSeconds":5,"maxStalenessPrefix":100},"databaseAccountOfferType":"Standard","defaultIdentity":"UserAssignedIdentity=/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kpumicsdevuws2umi","disableKeyBasedMetadataWriteAccess":false,"disableLocalAuth":false,"enableAnalyticalStorage":false,"enableAutomaticFailover":false,"enableBurstCapacity":false,"enableFreeTier":false,"enableMultipleWriteLocations":false,"enablePartitionMerge":false,"ipRules":[],"isVirtualNetworkFilterEnabled":false,"keyVaultKeyUri":"https://key_vault_name.vault.azure.net/keys/cosmos_name","locations":[{"failoverPriority":0,"id":"cosmos_name-westus2","isZoneRedundant":true,"locationName":"West US 2"}],"minimalTlsVersion":"Tls12","networkAclBypass":"None","networkAclBypassResourceIds":[],"publicNetworkAccess":"Disabled","virtualNetworkRules":[]},"tags":{"source":"terraform","version":"v1.0.1"}}
: timestamp=2024-11-22T21:38:30.561Z
2024-11-22T21:38:30.561Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: PUT https://management.azure.com/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15: timestamp=2024-11-22T21:38:30.561Z
2024-11-22T21:38:31.212Z [DEBUG] provider.terraform-provider-azurerm_v3.117.0_x5: AzureRM Response for https://management.azure.com/subscriptions/subscription_id/resourceGroups/resource_group/providers/Microsoft.DocumentDB/databaseAccounts/cosmos_name?api-version=2024-05-15:
HTTP/2.0 400 Bad Request
Content-Length: 283
Cache-Control: no-store, no-cache
Content-Type: application/json
Date: Fri, 22 Nov 2024 21:38:30 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: c74512b4-e300-c630-5586-b649468bf01d
X-Ms-Gatewayversion: version=2.14.0
X-Ms-Ratelimit-Remaining-Subscription-Global-Writes: 2999
X-Ms-Ratelimit-Remaining-Subscription-Writes: 199
X-Ms-Request-Id: eac6f596-a6ec-45fb-856b-73d875fc624a
X-Ms-Routing-Request-Id: WESTUS:20241122T213831Z:eac6f596-a6ec-45fb-856b-73d875fc624a
X-Msedge-Ref: Ref A: 37CA240E0571445294933656841D4455 Ref B: SJC211051203037 Ref C: 2024-11-22T21:38:30Z
{"code":"BadRequest","message":"The given default identity for cosmos_name is not valid. The default identity points to an user identity that does not exist in cosmos_name.\r\nActivityId: f244f3b4-14fe-45b6-8eaa-325384e37952, Microsoft.Azure.Documents.Common/2.14.0"}
: timestamp=2024-11-22T21:38:31.212Z
2024-11-22T21:38:31.213Z [ERROR] provider.terraform-provider-azurerm_v3.117.0_x5: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" diagnostic_severity=ERROR tf_resource_type=azurerm_cosmosdb_account tf_rpc=ApplyResourceChange @module=sdk.proto
diagnostic_summary=
| updating Database Account (Subscription: "subscription_id"
| Resource Group Name: "resource_group"
| Database Account Name: "cosmos_name"): creating/updating CosmosDB Account "cosmos_name" (Resource Group "resource_group"): performing DatabaseAccountsCreateOrUpdate: unexpected status 400 (400 Bad Request) with response: {"code":"BadRequest","message":"The given default identity for cosmos_name is not valid. The default identity points to an user identity that does not exist in cosmos_name.\r\nActivityId: f244f3b4-14fe-45b6-8eaa-325384e37952, Microsoft.Azure.Documents.Common/2.14.0"}
tf_provider_addr=provider tf_proto_version=5.6 tf_req_id=8f19d5d7-db0c-ddad-56b7-74450c63b138 timestamp=2024-11-22T21:38:31.213Z
Is there an existing issue for this?
Community Note
Terraform Version
1.7.5
AzureRM Provider Version
3.117.0, 4.11.0
Affected Resource(s)/Data Source(s)
azurerm_cosmosdb_account
Terraform Configuration Files
Debug Output/Panic Output
Expected Behaviour
The User Assigned Managed Identity should be applied first, then DefaultIdentity applied second to make it effective.
Actual Behaviour
400 Bad Error Response
Steps to Reproduce
FirstPartyIdentity
(default value)default_identity
terraform apply
failsImportant Factoids
No response
References
Default Identity is applied first: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/services/cosmos/cosmosdb_account_resource.go#L1134-L1146
UMI is applied second, DefaultIdentity is fixed up: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/services/cosmos/cosmosdb_account_resource.go#L1301-L1330
Original Issue: #22466