VMSS extension protected setting encoding/json causes re-creates

websolut commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

v1.5.7

AzureRM Provider Version

3.57

Affected Resource(s)/Data Source(s)

azurerm_windows_virtual_machine_scale_set

Terraform Configuration Files

We deploy service fabric clusters (windows). We install a number of extensions including:

  extension {

    name                 = "DSCExtension"
    publisher            = "Microsoft.Powershell"
    type                 = "DSC"
    type_handler_version = "2.9"
    force_update_tag     = var.dsc_force_update_tag_version

    settings = jsonencode({
      "configuration" : {
        "url" : "${data.azurerm_storage_blob.dsc.url}",
        "script" : "${var.dsc_script_name}.ps1",
        "function" : "Main"
      },
      "configurationArguments" : {
        "nodeName" : "localhost"
      }
    })

    protected_settings = jsonencode({
      "configurationUrlSasToken" = "${var.dsc_storage_account_sas_token}"
    }) 

  }

Debug Output/Panic Output

Occasionally we observed this TF plan:
# module.service_fabric.scale_set["1"].azurerm_windows_virtual_machine_scale_set.scale_set will be updated in-place
  ~ resource "azurerm_windows_virtual_machine_scale_set" "scale_set" {
        id                                                = "/subscriptions/**/resourceGroups/dev/providers/Microsoft.Compute/virtualMachineScaleSets/ss-1"
        name                                              = "ss-1"
        tags                                              = {}
        # (25 unchanged attributes hidden)

      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      - extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }
      + extension {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }

Then it takes 5-6 hours of extension recreation.

Expected Behaviour

Extensions should not be reinstalled.

Actual Behaviour

We noticed: We get the SAS token from a key vault. In the state we can see it in a json format (with some escapes): "value": "?sv=2018-11-09\u0026sr=c\u0026st=2023-08-29T15:10:38Z\u0026se=2024-09-01T15:10:38Z\u0026sp=racwdl\u0026spr=https\u0026sig=R**",

Extension state we see even more escapes: "protected_settings": "{\"configurationUrlSasToken\":\"?sv=2018-11-09\\u0026sr=c\\u0026st=2023-08-29T15:10:38Z\\u0026se=2024-09-01T15:10:38Z\\u0026sp=racwdl\\u0026spr=https\\u0026sig=R**\"}",

What we noticed with those strange TF apply runs is that state files for extensions had some additional encodings (%3A instead of ':' ): "protected_settings": "{\"configurationUrlSasToken\":\"?sv=2018-11-09\\u0026sr=c\\u0026st=2023-08-29T15%3A10%3A38Z\\u0026se=2024-09-01T15%3A10%3A38Z\\u0026sp=racwdl\\u0026spr=https\\u0026sig=R**\"}",

We had to manually edit the state file with a lot of environments to make sure that TF does not envisage any changes related to this extension.

Steps to Reproduce

We are not sure about why this issue appears. We only use Azure devops to run TF.

Important Factoids

No response

References

No response

websolut commented 1 year ago

I apologize I will close this ticket as it seems the issue is related to azurerm provider upgrade (SAS token was generated differently) with a newer version.

github-actions[bot] commented 5 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm