search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.54k stars 4.61k forks source link

User Assigned Identity Removal Ineffective in VMSS Identity Block #25058

Open saketh-cloudknox opened 7 months ago

saketh-cloudknox commented 7 months ago

Is there an existing issue for this?

Community Note

Terraform Version

1.2.1

AzureRM Provider Version

3.93.0

Affected Resource(s)/Data Source(s)

azurerm_linux_virtual_machine_scale_set

Terraform Configuration Files

# Reference: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/linux_virtual_machine_scale_set

terraform {
  required_version = ">=0.12"
}

locals {
  first_public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+wWK73dCr+jgQOAxNsHAnNNNMEMWOHYEccp6wJm2gotpr9katuF/ZAdou5AaW1C61slRkHRkpRRX9FA9CYBiitZgvCCz+3nWNN7l/Up54Zps/pHWGZLHNJZRYyAB6j5yVLMVHIHriY49d/GZTZVNB8GoJv9Gakwc/fuEZYYl4YDFiGMBP///TzlI4jhiJzjKnEvqPFki5p2ZRJqcbCiF4pJrxUQR/RXqVFQdbRLZgYfJ8xGB878RENq3yQ39d8dVOkq4edbkzwcUmwwwkYVPIoDGsYLaRHnG+To7FvMeyO7xDVQkMKzopTQV8AuKpyvpqu0a9pWOMaiCyDytO7GGN you@me.com"
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West US"
}

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "internal" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]
}

resource "azurerm_user_assigned_identity" "vmss_identity_1" {
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  name                = "test-vmss-identity-1"
}

resource "azurerm_user_assigned_identity" "vmss_identity_2" {
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  name                = "test-vmss-identity-2"
}

resource "azurerm_linux_virtual_machine_scale_set" "example" {
  name                = "example-vmss"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "Standard_F2"
  instances           = 1
  admin_username      = "adminuser"

  identity {
    type         = "UserAssigned"
    # I first created the identity_ids with 2 identities and it created the both and asssociated with the VMSS, then I ran it with just 1 to see if it will unassign 1 and it did not. 
    identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id , azurerm_user_assigned_identity.vmss_identity_2.id]
    # identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id]
  }

  admin_ssh_key {
    username   = "adminuser"
    public_key = local.first_public_key
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  os_disk {
    storage_account_type = "Standard_LRS"
    caching              = "ReadWrite"
  }

  network_interface {
    name    = "example"
    primary = true

    ip_configuration {
      name      = "internal"
      primary   = true
      subnet_id = azurerm_subnet.internal.id
    }
  }
}

Debug Output/Panic Output

https://gist.github.com/saketh-cloudknox/7b78170917f119c259cc16cc7a6428f6

Expected Behaviour

The user assigned identity that was removed from the identity block should have been removed from the VMSS.

Actual Behaviour

The user assigned identity that was removed from the identity block remained attached to the VMSS.

Steps to Reproduce

User Assigned Identity Removal Ineffective in VMSS Identity Block

Reproduction Steps

  1. Run terraform apply --auto-approve. Terraform shows that 2 User assigned identities will be assigned to the VMSS. This is reflected in the Azure Portal.
  2. Remove one of the identity_ids from the identity block under the azurerm_linux_virtual_machine_scale_set.
  3. Run terraform apply --auto-approve. Terraform shows that 1 User assigned identity will be unassigned from the VMSS as per the plan.
  4. Removal is not reflected in the azure portal or via CLI. VMSS still has BOTH user assigned identities associated with it.

Important Factoids

Azure Production

References

No response

saketh-cloudknox commented 7 months ago

Azure VMSS Identity Update Terraform Issues

Hi all, I did some digging today into why this is happening and I think this should summarize the issue and the potential fix. Looking to get some opinions on it!

1. Problem Statement

The azurerm Terraform Provider does not properly remove identities from the VMSS when removing them from the Terraform configuration. The Terraform Provider tries to patch the VMSS using only the identities that are present in the Terraform configuration. When existing identities are patched with the same values, the API does not remove any identities not present in the PATCH request. The identities are passed in as a map where the key is the identity and identities to be removed have their value set to null, while identities to be kept/added are set to {}

1.1. REST API Behavior

This request will remove the userAssignedIdentity with the id /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2. It will not remove any other identities that are present in the VMSS.

PATCH https://management.azure.com/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-09-01
{
    "identity": {
        "type": "userassigned",
        "userAssignedIdentities": {
            "/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2": null
        }
    }
}

This request will not remove any identities from the VMSS, even though the identity with the id /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2 is not present in the PATCH request.

PATCH https://management.azure.com/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-09-01
{
    "identity": {
        "type": "userassigned",
        "userAssignedIdentities": {
            "/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-1": {}
        }
    }
}

1.2 Terraform Provider Behavior

If the initial state of the VMSS looks like the following in terraform, then terraform will ensure that the VMSS has both idenitites attached to it.

resource "azurerm_linux_virtual_machine_scale_set" "example" {
  name                = "example-vmss"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "Standard_F2"
  instances           = 1
  admin_username      = "adminuser"

  identity {
    type         = "UserAssigned" 
    identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id , azurerm_user_assigned_identity.vmss_identity_2.id]
  }

  admin_ssh_key {
    username   = "adminuser"
    public_key = local.first_public_key
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  os_disk {
    storage_account_type = "Standard_LRS"
    caching              = "ReadWrite"
  }

  network_interface {
    name    = "example"
    primary = true

    ip_configuration {
      name      = "internal"
      primary   = true
      subnet_id = azurerm_subnet.internal.id
    }
  }
}

The user then removes the identity with the id azurerm_user_assigned_identity.vmss_identity_2.id from the VMSS configuration and runs terraform apply.

resource "azurerm_linux_virtual_machine_scale_set" "example" {
  name                = "example-vmss"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "Standard_F2"
  instances           = 1
  admin_username      = "adminuser"

  identity {
    type         = "UserAssigned" 
    identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id]
  }

  admin_ssh_key {
    username   = "adminuser"
    public_key = local.first_public_key
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  os_disk {
    storage_account_type = "Standard_LRS"
    caching              = "ReadWrite"
  }

  network_interface {
    name    = "example"
    primary = true

    ip_configuration {
      name      = "internal"
      primary   = true
      subnet_id = azurerm_subnet.internal.id
    }
  }
}

The terraform provider will then patch the VMSS with the following request (according to the debug logs).

PATCH /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-03-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.21.6 (amd64-darwin) go-autorest/v14.2.1 tombuildsstuff/kermit/v0.20240122.1123108 compute/2023-03-01 HashiCorp Terraform/1.2.1 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.93.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 446
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 0794a696-9d83-f3e7-07c6-d6569402a777
Accept-Encoding: gzip

{
    "identity": {
        "type": "UserAssigned",
        "userAssignedIdentities": {
            "/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-1": {}
        }
    },
    "properties": {
        "upgradePolicy": {
            "mode": "Manual"
        },
        "virtualMachineProfile": {
            "storageProfile": {
                "imageReference": {
                    "offer": "0001-com-ubuntu-server-jammy",
                    "publisher": "Canonical",
                    "sku": "22_04-lts",
                    "version": "latest"
                }
            }
        }
    }
}

Based on the API behvaior noted in section 1.1, the VMSS will still have the identity with the id azurerm_user_assigned_identity.vmss_identity_2.id attached to it since it did not set the identity to null in the PATCH request and only set the identity with the id azurerm_user_assigned_identity.vmss_identity_1.id to {}.

2. Proposed Solution

2.1. Update Terraform Provider

In /internal/services/compute/linux_virtual_machine_scale_set_resource.go the Update method should be updated to remove any identities that are not present in the Terraform configuration.

The Update method associated with identities currently has this code block:

if d.HasChange("identity") {
  identityRaw := d.Get("identity").([]interface{})
  identity, err := expandVirtualMachineScaleSetIdentity(identityRaw)
  if err != nil {
    return fmt.Errorf("expanding `identity`: %+v", err)
  }

  update.Identity = identity
}

It is not adding the entry in the identity map for the identities that need to be removed from the VMSS by setting their value to null. The expandVirtualMachineScaleSetIdentity function should be updated to add the identities that need to be removed to the map with a value of null. These identities are available via the variable existing.Identity.UserAssignedIdentities which will give the existing identities that can be compared against the identities returned by expandVirtualMachineScaleSetIdentity

ms-zhenhua commented 7 months ago

Hi @saketh-cloudknox, thank you for reporting this issue. However, it may not be fixed at once because I found in some situation the backend will reply an error for the identity with null value. We need to double confirm with Azure team about the reason. 

PATCH
X-Ms-Correlation-Request-Id: e94fd18c-905c-a5e5-5c1c-566f79e19741
{
    "identity": {
        "type": "UserAssigned",
        "userAssignedIdentities": {
            "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/acctestRG-OVMSS-240305172229758653/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acctest2oxyl2": null,
            "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/acctestRG-OVMSS-240305172229758653/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acctestoxyl2": {}
        }
    },
    "properties": {
        "virtualMachineProfile": {
            "osProfile": {
                "linuxConfiguration": {}
            },
            "storageProfile": {
                "imageReference": {
                    "offer": "0001-com-ubuntu-server-jammy",
                    "publisher": "Canonical",
                    "sku": "22_04-lts",
                    "version": "latest"
                }
            }
        }
    }
}

RESPONSE
{
  "error": {
    "code": "InvalidParameter",
    "message": "ResourceIdentity userAssignedIdentities should not have keys with null values.",
    "target": "resourceIdentity.userAssignedIdentities"
  }
}

Currently, there is a tricky way to temporarily unblock this issue: remove azurerm_user_assigned_identity.vmss_identity_2 block with azurerm_user_assigned_identity.vmss_identity_2.id of azurerm_linux_virtual_machine_scale_set.example together.

tombuildsstuff commented 7 months ago

@ms-zhenhua

I found in some situation the backend will reply an error for the identity with null value.

This is the expected API behaviour FWIW, you must specify an empty payload rather than null - it's a consistent behaviour across multiple Azure Services