azurerm_linux_virtual_machine_scale_set: Case sensitivity issues with source_image_id and identity_ids

Nirusu commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.3.1

AzureRM Provider Version

3.26.0

Affected Resource(s)/Data Source(s)

azurerm_linux_virtual_machine_scale_set

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.26.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.4.1"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "random_password" "password" {
  length = 16
}

resource "azurerm_linux_virtual_machine_scale_set" "scale_set" {
  name                            = "test"
  resource_group_name             = "test"
  location                        = "westeurope"
  sku                             = "Standard_DC4as_v5"
  instances                       = 1
  admin_username                  = "adminuser"
  admin_password                  = random_password.password.result
  overprovision                   = false
  vtpm_enabled                    = true
  disable_password_authentication = false
  upgrade_mode                    = "Manual"
  secure_boot_enabled             = true
  source_image_id                 = "/CommunityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df/Images/constellation/Versions/2.1.0"

  identity {
    type         = "UserAssigned"
    identity_ids = ["/subscriptions/0d202bbb-4fa7-4af8-8125-58c269a05435/resourcegroups/xxxxxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/yyyyy"]
  }

  boot_diagnostics {}

  os_disk {
    security_encryption_type = "VMGuestStateOnly"
    caching                  = "ReadWrite"
    storage_account_type     = "Premium_LRS"
  }

  network_interface {
    name    = "node-network"
    primary = true

    ip_configuration {
      name    = "node-network"
      primary = true
    }
  }
}

Debug Output/Panic Output

https://gist.github.com/Nirusu/dbced209355480a1f9bfcb9374cf29bb

Expected Behaviour

Terraform is able to parse the source_image_id and identity_ids with the casing returned from the Azure API, as these resources seem to be case-insensitive by or to the API.

Actual Behaviour

Terraform fails to create a plan / apply unless the casing is changed in the following ways:

source_image_id: /communityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df/images/constellation/versions/2.1.0 (had to change CommunityGalleries ->. communityGalleries, Images -> images, Versions -> versions)
identity_ids: /subscriptions/0d202bbb-4fa7-4af8-8125-58c269a05435/resourceGroups/xxxxxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/yyyyy (had to change resourcegroups -> resourceGroups)

After changing the casing for these types, Terraform is able to create a plan and also apply the infrastructure (at least when using the full version, haven't tested the minimal one to this point).

Steps to Reproduce

terraform apply

This should already trigger the parsing errors.

The URLs triggering the issue were mostly taken from Azure CLI outputs directly. The CommunityGalleries one with CamelCase is returned by az sig list-community:

$ az sig list-community | grep ConstellationCVM
[...snip...]
    "id": "/providers/Microsoft.Compute/locations/eastus/CommunityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df",
    "name": "ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df",
    "uniqueId": "/CommunityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df"
[...snip...]

The ManagedIdentity with lowercase resourcegroups is returned as-is by az identity list:

az identity list -g "xxxxxxxx"
[
  {
    "clientId": "zzzzzzzzzzzzzzz",
    "id": "/subscriptions/0d202bbb-4fa7-4af8-8125-58c269a05435/resourcegroups/xxxxxxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/yyyyyyyy,
    "location": "northeurope",
     [...snip...]
    "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
  }
]

Important Factoids

Example uses Confidential VMs (and Community Gallery specifically expects confidential VMs)

References

Detected as part of: https://github.com/edgelesssys/constellation/pull/233 Also has the full Terraform example files and not just the reduced version presented here.

myc2h6o commented 2 years ago

HI @Nirusu thanks for opening the issue! Sorry for the inconvenience, but unfortunately it is by design in the AzureRM provider to restrict the casing. When a referenced resource ID is coming from outside of Terraform, it needs to match the ID format defined in the provider. And for Terraform managed resources, e.g. getting the Identity ID from azurerm_user_assigned_identity, the ID is taken care of by the ID formatter

Nirusu commented 2 years ago

But why is it by design if the Azure API seemingly doesn't care about case sensitivity? Are there actually endpoints where the casing does matter to the API?

myc2h6o commented 2 years ago

I've done some search and found this comment in another issue: https://github.com/hashicorp/terraform-provider-azurerm/issues/1386#issuecomment-404467465

In addition whilst Terraform is also generally case sensitive (although we're a bit more lenient for Enum values in the Azure Provider than most other Providers since the API's are generally case insensitive for Enum values) - in regard to ID's we rely on the casing of the URI's being correct when we parse segments out of the URI, whilst we could update this - I believe taking this approach would lead to additional complexity inside the Provider whilst making the user experience inconsistent since this is inconsistent across API's.

Azure API accepts different casing, however the value returned by the API sometimes don't treat the case in the ID very well, in order to make the user experience consistent, the provider restricts the casing of fixed segments in the ID.

tombuildsstuff commented 2 years ago

@Nirusu

But why is it by design if the Azure API seemingly doesn't care about case sensitivity? Are there actually endpoints where the casing does matter to the API?

Yes, and unfortunately this can be really subtle - for example some clusters get provisioned successfully but you can't log-into them - in others we've seen data not flow correctly - it's infrequent but when it pops up it's problematic, also as @myc2h6o has mentioned normalizing this to a single expected casing has some UX benefits (insofar as we can output exactly which Resource ID we're expecting).

In this particular case we should be normalizing these values in the Read functions so that we're parsing whatever casing the API is returning (using the Parse{Name}Insensitive function) so that these are consistent within the provider, @myc2h6o would you be able to take a look into fixing that?

Thanks!

myc2h6o commented 2 years ago

@tombuildsstuff The Read/Flatten functions already did that. It doesn't validate the source_image_id, and use Parse{Name}Insensitive for identity_ids. The original error seems to be happening in the schema validation so the error is expected?

hashicorp / terraform-provider-azurerm