Open saketh-cloudknox opened 8 months ago
Hi all, I did some digging today into why this is happening and I think this should summarize the issue and the potential fix. Looking to get some opinions on it!
The azurerm
Terraform Provider does not properly remove identities from the VMSS when removing them from the Terraform configuration. The Terraform Provider tries to patch the VMSS using only the identities that are present in the Terraform configuration. When existing identities are patched with the same values, the API does not remove any identities not present in the PATCH request. The identities are passed in as a map where the key is the identity and identities to be removed have their value set to null
, while identities to be kept/added are set to {}
This request will remove the userAssignedIdentity with the id /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2
. It will not remove any other identities that are present in the VMSS.
PATCH https://management.azure.com/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-09-01
{
"identity": {
"type": "userassigned",
"userAssignedIdentities": {
"/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2": null
}
}
}
This request will not remove any identities from the VMSS, even though the identity with the id /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2
is not present in the PATCH request.
PATCH https://management.azure.com/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-09-01
{
"identity": {
"type": "userassigned",
"userAssignedIdentities": {
"/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-1": {}
}
}
}
If the initial state of the VMSS looks like the following in terraform, then terraform will ensure that the VMSS has both idenitites attached to it.
resource "azurerm_linux_virtual_machine_scale_set" "example" {
name = "example-vmss"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "Standard_F2"
instances = 1
admin_username = "adminuser"
identity {
type = "UserAssigned"
identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id , azurerm_user_assigned_identity.vmss_identity_2.id]
}
admin_ssh_key {
username = "adminuser"
public_key = local.first_public_key
}
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts"
version = "latest"
}
os_disk {
storage_account_type = "Standard_LRS"
caching = "ReadWrite"
}
network_interface {
name = "example"
primary = true
ip_configuration {
name = "internal"
primary = true
subnet_id = azurerm_subnet.internal.id
}
}
}
The user then removes the identity with the id azurerm_user_assigned_identity.vmss_identity_2.id
from the VMSS configuration and runs terraform apply
.
resource "azurerm_linux_virtual_machine_scale_set" "example" {
name = "example-vmss"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "Standard_F2"
instances = 1
admin_username = "adminuser"
identity {
type = "UserAssigned"
identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id]
}
admin_ssh_key {
username = "adminuser"
public_key = local.first_public_key
}
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts"
version = "latest"
}
os_disk {
storage_account_type = "Standard_LRS"
caching = "ReadWrite"
}
network_interface {
name = "example"
primary = true
ip_configuration {
name = "internal"
primary = true
subnet_id = azurerm_subnet.internal.id
}
}
}
The terraform provider will then patch the VMSS with the following request (according to the debug logs).
PATCH /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-03-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.21.6 (amd64-darwin) go-autorest/v14.2.1 tombuildsstuff/kermit/v0.20240122.1123108 compute/2023-03-01 HashiCorp Terraform/1.2.1 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.93.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 446
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 0794a696-9d83-f3e7-07c6-d6569402a777
Accept-Encoding: gzip
{
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-1": {}
}
},
"properties": {
"upgradePolicy": {
"mode": "Manual"
},
"virtualMachineProfile": {
"storageProfile": {
"imageReference": {
"offer": "0001-com-ubuntu-server-jammy",
"publisher": "Canonical",
"sku": "22_04-lts",
"version": "latest"
}
}
}
}
}
Based on the API behvaior noted in section 1.1, the VMSS will still have the identity with the id azurerm_user_assigned_identity.vmss_identity_2.id
attached to it since it did not set the identity to null
in the PATCH request and only set the identity with the id azurerm_user_assigned_identity.vmss_identity_1.id
to {}
.
In /internal/services/compute/linux_virtual_machine_scale_set_resource.go
the Update
method should be updated to remove any identities that are not present in the Terraform configuration.
The Update
method associated with identities currently has this code block:
if d.HasChange("identity") {
identityRaw := d.Get("identity").([]interface{})
identity, err := expandVirtualMachineScaleSetIdentity(identityRaw)
if err != nil {
return fmt.Errorf("expanding `identity`: %+v", err)
}
update.Identity = identity
}
It is not adding the entry in the identity map for the identities that need to be removed from the VMSS by setting their value to null
. The expandVirtualMachineScaleSetIdentity
function should be updated to add the identities that need to be removed to the map with a value of null
. These identities are available via the variable existing.Identity.UserAssignedIdentities
which will give the existing identities that can be compared against the identities returned by expandVirtualMachineScaleSetIdentity
Hi @saketh-cloudknox, thank you for reporting this issue. However, it may not be fixed at once because I found in some situation the backend will reply an error for the identity with null
value. We need to double confirm with Azure team about the reason.
PATCH
X-Ms-Correlation-Request-Id: e94fd18c-905c-a5e5-5c1c-566f79e19741
{
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/acctestRG-OVMSS-240305172229758653/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acctest2oxyl2": null,
"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/acctestRG-OVMSS-240305172229758653/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acctestoxyl2": {}
}
},
"properties": {
"virtualMachineProfile": {
"osProfile": {
"linuxConfiguration": {}
},
"storageProfile": {
"imageReference": {
"offer": "0001-com-ubuntu-server-jammy",
"publisher": "Canonical",
"sku": "22_04-lts",
"version": "latest"
}
}
}
}
}
RESPONSE
{
"error": {
"code": "InvalidParameter",
"message": "ResourceIdentity userAssignedIdentities should not have keys with null values.",
"target": "resourceIdentity.userAssignedIdentities"
}
}
Currently, there is a tricky way to temporarily unblock this issue: remove azurerm_user_assigned_identity.vmss_identity_2
block with azurerm_user_assigned_identity.vmss_identity_2.id
of azurerm_linux_virtual_machine_scale_set.example
together.
@ms-zhenhua
I found in some situation the backend will reply an error for the identity with null value.
This is the expected API behaviour FWIW, you must specify an empty payload rather than null
- it's a consistent behaviour across multiple Azure Services
Is there an existing issue for this?
Community Note
Terraform Version
1.2.1
AzureRM Provider Version
3.93.0
Affected Resource(s)/Data Source(s)
azurerm_linux_virtual_machine_scale_set
Terraform Configuration Files
Debug Output/Panic Output
Expected Behaviour
The user assigned identity that was removed from the
identity
block should have been removed from the VMSS.Actual Behaviour
The user assigned identity that was removed from the
identity
block remained attached to the VMSS.Steps to Reproduce
User Assigned Identity Removal Ineffective in VMSS Identity Block
Reproduction Steps
terraform apply --auto-approve
. Terraform shows that 2 User assigned identities will be assigned to the VMSS. This is reflected in the Azure Portal.identity_ids
from theidentity
block under theazurerm_linux_virtual_machine_scale_set
.terraform apply --auto-approve
. Terraform shows that 1 User assigned identity will be unassigned from the VMSS as per the plan.Important Factoids
Azure Production
References
No response