Closed jkroepke closed 3 weeks ago
Seems like a breaking change in 1.13. Body now requires a real HCL structure instead JSON
The error persists even after migrate the code.
Hi @jkroepke ,
Thank you for taking time to open this issue.
The v1.13.x introduces the dynamic properties which allow user to use a real HCL, but the old JSON solution still works, so it's not required to migrate.
The error message comes from the terraform cli: https://github.com/hashicorp/terraform/blob/main/internal/terraform/evaluate.go#L742 and it states "this should not happen" 🤔
Would you please share the terraform version that you're using? And would you try the latest terraform?
And I have a workaround that might fix it:
terraform plan
command, it's expected to see there's no changes.terraform refresh
command to refresh the terraform state.I just upgrade from 1.12.1 to 1.13.1 and it fails at step 2.
I got this error in both situation, where I don't touch the code and where I apply the migration to dynamic HCL.
Maybe the error is gallery application version specific?
Downgrade to 1.12.1, plan and apply works fine.
Thanks for trying the workaround.
Maybe the error is gallery application version specific?
Yes, it seems so.
Would you please share me the config to reproduce this issue? Thanks
the resource is extremely integrated into our module.
Maybe this is sufficient.
resource "azapi_resource" "gallery_application_version_windows" {
for_each = toset(var.installer_versions)
type = "Microsoft.Compute/galleries/applications/versions@2022-03-03"
name = each.key
parent_id = azurerm_gallery_application.windows.id
location = azurerm_gallery_application.windows.location
locks = [azurerm_gallery_application.windows.id]
tags = var.tags
body = nonsensitive(jsonencode({
properties = {
publishingProfile = {
enableHealthCheck = false
customActions = [
{
name = "opsstack-agent-reinstall"
description = "Running the installer again"
script = "cmd.exe /c .\\${azurerm_gallery_application.windows.name}.cmd"
}
]
manageActions = {
install = "cmd.exe /c .\\${azurerm_gallery_application.windows.name}.cmd"
remove = "ls"
update = "cmd.exe /c .\\${azurerm_gallery_application.windows.name}.cmd"
}
replicationMode = "Full"
replicaCount = 1
settings = {
packageFileName = "${azurerm_gallery_application.windows.name}.cmd"
}
source = {
mediaLink = "${azurerm_storage_blob.windows[each.key].id}${data.azurerm_storage_account_blob_container_sas.this.sas}"
}
storageAccountType = "Standard_ZRS"
targetRegions = flatten([
for _, region in concat(
[
{
region = azurerm_gallery_application.windows.location
replicas = 1
}
], var.opsstack_agent_additional_regions
) :
[
{
name = local.location_cli_mapping[region.region]
regionalReplicaCount = region.replicas
storageAccountType = "Standard_ZRS"
}
]
]),
},
safetyProfile = {
allowDeletionOfReplicatedLocations = true
}
}
}))
ignore_body_changes = [
"properties.publishingProfile.source.media_link"
]
response_export_values = ["properties.provisioningState"]
lifecycle {
replace_triggered_by = [
azurerm_storage_blob.linux[each.key].id
]
postcondition {
condition = jsondecode(self.output).properties.provisioningState == "Succeeded"
error_message = "${self.name} returned an provisioningState != ${jsondecode(self.output).properties.provisioningState}"
}
}
}
Thanks for the config, I tried to use this config to reproduce it, but failed. Here's my complete config which is created by v1.12.1 and is able to upgrade to v1.13.0. And I'm using the latest Terraform version.
Hope it could help you.
resource "azurerm_resource_group" "test" {
name = "acctest-compute-henglu"
location = "westeurope"
}
resource "azurerm_shared_image_gallery" "test" {
name = "henglu429"
resource_group_name = azurerm_resource_group.test.name
location = azurerm_resource_group.test.location
}
resource "azurerm_gallery_application" "test" {
name = "henglu429"
gallery_id = azurerm_shared_image_gallery.test.id
location = azurerm_resource_group.test.location
supported_os_type = "Windows"
}
resource "azurerm_storage_account" "test" {
name = "henglu429"
resource_group_name = azurerm_resource_group.test.name
location = azurerm_resource_group.test.location
account_tier = "Standard"
account_replication_type = "LRS"
}
resource "azurerm_storage_container" "test" {
name = "henglu429"
storage_account_name = azurerm_storage_account.test.name
container_access_type = "blob"
}
resource "azurerm_storage_blob" "test" {
name = "scripts"
storage_account_name = azurerm_storage_account.test.name
storage_container_name = azurerm_storage_container.test.name
type = "Page" # Use Page Blob as a workaround to UnmanagedStorageAccount quota issue
size = 512
}
data "azurerm_storage_account_blob_container_sas" "test" {
connection_string = "${azurerm_storage_account.test.primary_connection_string}"
container_name = "${azurerm_storage_container.test.name}"
https_only = true
start = "2017-03-21"
expiry = "2022-03-21"
permissions {
read = true
add = false
create = false
write = true
delete = false
list = true
}
}
resource "azapi_resource" "gallery_application_version_windows" {
type = "Microsoft.Compute/galleries/applications/versions@2022-03-03"
name = "0.0.1"
parent_id = azurerm_gallery_application.test.id
location = azurerm_gallery_application.test.location
locks = [azurerm_gallery_application.test.id]
body = nonsensitive(jsonencode({
properties = {
publishingProfile = {
enableHealthCheck = false
customActions = [
{
name = "opsstack-agent-reinstall"
description = "Running the installer again"
script = "cmd.exe /c .\\${azurerm_gallery_application.test.name}.cmd"
}
]
manageActions = {
install = "cmd.exe /c .\\${azurerm_gallery_application.test.name}.cmd"
remove = "ls"
update = "cmd.exe /c .\\${azurerm_gallery_application.test.name}.cmd"
}
replicationMode = "Full"
replicaCount = 1
settings = {
packageFileName = "${azurerm_gallery_application.test.name}.cmd"
}
source = {
mediaLink = azurerm_storage_blob.test.id
}
storageAccountType = "Standard_ZRS"
targetRegions = [
{
name = "West Europe"
regionalReplicaCount = 1
storageAccountType = "Standard_ZRS"
}
]
},
safetyProfile = {
allowDeletionOfReplicatedLocations = true
}
}
}))
ignore_body_changes = [
"properties.publishingProfile.source.media_link"
]
response_export_values = ["properties.provisioningState"]
}
Is it possible to remove them from the state and import them by v1.13.0? Though I guess it might not.
The only different I see is that azurerm_storage_account_blob_container_sas
is not used on the mediaLink properties and I used for_each
to generated multiple. But I guess it wont change anything.
I mean almost two days of work to do that. Having 13 VM applications across 15 customers deployments. Based on how VM Applications are working, they can't be delete in parallel and it takes approx. 5 up to 10 minutes per version to remove.
Its also mentionable that I got the error for the same resource multiple times:
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.3"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.3"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│
│ on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│ 106: resource "azapi_resource" "gallery_application_version_windows" {
│
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.3"]
│ data could not be decoded from the state: missing expected {.
If you want, you can share your state from your test (after apply 1.12) and I can compare your state with my state and look for differences.
Thanks for the details!
And I'd like to double confirm your terraform version. Recently I saw some issues are resolved by the latest terraform version.
Here're the states for v1.12.1 and v1.13.1. I've redacted all the sensitive data. Hope it could help.
v1.12.1:
{
"mode": "managed",
"type": "azapi_resource",
"name": "gallery_application_version_windows",
"provider": "provider[\"registry.terraform.io/azure/azapi\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"body": "{\"properties\":{\"publishingProfile\":{\"customActions\":[{\"description\":\"Running the installer again\",\"name\":\"opsstack-agent-reinstall\",\"script\":\"cmd.exe /c .\\\\example.cmd\"}],\"enableHealthCheck\":false,\"manageActions\":{\"install\":\"cmd.exe /c .\\\\example.cmd\",\"remove\":\"ls\",\"update\":\"cmd.exe /c .\\\\example.cmd\"},\"replicaCount\":1,\"replicationMode\":\"Full\",\"settings\":{\"packageFileName\":\"example.cmd\"},\"source\":{\"mediaLink\":\"https://example.blob.core.windows.net/example/scripts\"},\"storageAccountType\":\"Standard_ZRS\",\"targetRegions\":[{\"name\":\"West Europe\",\"regionalReplicaCount\":1,\"storageAccountType\":\"Standard_ZRS\"}]},\"safetyProfile\":{\"allowDeletionOfReplicatedLocations\":true}}}",
"id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example/versions/0.0.1",
"identity": [],
"ignore_body_changes": [
"properties.publishingProfile.source.media_link"
],
"ignore_casing": false,
"ignore_missing_property": true,
"location": "westeurope",
"locks": [
"/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example"
],
"name": "0.0.1",
"output": "{\"properties\":{\"provisioningState\":\"Succeeded\"}}",
"parent_id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example",
"removing_special_chars": false,
"response_export_values": [
"properties.provisioningState"
],
"schema_validation_enabled": true,
"tags": {},
"timeouts": null,
"type": "Microsoft.Compute/galleries/applications/versions@2022-03-03"
},
"sensitive_attributes": [],
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJkZWxldGUiOjE4MDAwMDAwMDAwMDAsInJlYWQiOjMwMDAwMDAwMDAwMCwidXBkYXRlIjoxODAwMDAwMDAwMDAwfX0=",
"dependencies": [
"azurerm_gallery_application.test",
"azurerm_resource_group.test",
"azurerm_shared_image_gallery.test",
"azurerm_storage_account.test",
"azurerm_storage_blob.test",
"azurerm_storage_container.test"
]
}
]
}
v1.13.1:
{
"mode": "managed",
"type": "azapi_resource",
"name": "gallery_application_version_windows",
"provider": "provider[\"registry.terraform.io/azure/azapi\"]",
"instances": [
{
"schema_version": 1,
"attributes": {
"body": {
"value": "{\"properties\":{\"publishingProfile\":{\"customActions\":[{\"description\":\"Running the installer again\",\"name\":\"opsstack-agent-reinstall\",\"script\":\"cmd.exe /c .\\\\example.cmd\"}],\"enableHealthCheck\":false,\"manageActions\":{\"install\":\"cmd.exe /c .\\\\example.cmd\",\"remove\":\"ls\",\"update\":\"cmd.exe /c .\\\\example.cmd\"},\"replicaCount\":1,\"replicationMode\":\"Full\",\"settings\":{\"packageFileName\":\"example.cmd\"},\"source\":{\"mediaLink\":\"https://example.blob.core.windows.net/example/scripts\"},\"storageAccountType\":\"Standard_ZRS\",\"targetRegions\":[{\"name\":\"West Europe\",\"regionalReplicaCount\":1,\"storageAccountType\":\"Standard_ZRS\"}]},\"safetyProfile\":{\"allowDeletionOfReplicatedLocations\":true}}}",
"type": "string"
},
"id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example/versions/0.0.1",
"identity": [],
"ignore_body_changes": [
"properties.publishingProfile.source.media_link"
],
"ignore_casing": false,
"ignore_missing_property": true,
"location": "westeurope",
"locks": [
"/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example"
],
"name": "0.0.1",
"output": {
"value": "{\"properties\":{\"provisioningState\":\"Succeeded\"}}",
"type": "string"
},
"parent_id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example",
"removing_special_chars": false,
"response_export_values": [
"properties.provisioningState"
],
"schema_validation_enabled": true,
"tags": {},
"timeouts": null,
"type": "Microsoft.Compute/galleries/applications/versions@2022-03-03"
},
"sensitive_attributes": [],
"dependencies": [
"azurerm_gallery_application.test",
"azurerm_resource_group.test",
"azurerm_shared_image_gallery.test",
"azurerm_storage_account.test",
"azurerm_storage_blob.test",
"azurerm_storage_container.test"
]
}
]
}
TF version: v1.8.2
I also noice that other azapi resource are note affected. At least I do not see any errors. But Terraform shows only 100 errors and the first 100 errors coming from the azapi gallery_application_version
Thats the state on my side:
{
"module": "module.opsstack",
"mode": "managed",
"type": "azapi_resource",
"name": "gallery_application_version_windows",
"provider": "provider[\"registry.terraform.io/azure/azapi\"]",
"instances": [
{
"index_key": "1.1.0",
"schema_version": 0,
"attributes": {
"body": "{\"properties\":{\"publishingProfile\":{\"customActions\":[{\"description\":\"Running the Ops.Stack Agent installer again\",\"name\":\"opsstack-agent-reinstall\",\"script\":\"cmd.exe /c .\\\\opsstack-agent-windows.cmd\"}],\"enableHealthCheck\":false,\"manageActions\":{\"install\":\"cmd.exe /c .\\\\opsstack-agent-windows.cmd\",\"remove\":\"ls\",\"update\":\"cmd.exe /c .\\\\opsstack-agent-windows.cmd\"},\"replicaCount\":1,\"replicationMode\":\"Full\",\"settings\":{\"packageFileName\":\"opsstack-agent-windows.cmd\"},\"source\":{\"mediaLink\":\"https://removed.blob.core.windows.net/opsstack-installer-scripts/windows/install1.1.0.cmd?sv=2018-11-09\\u0026sr=c\\u0026st=2020-01-01\\u0026se=2040-01-01\\u0026sp=rl\\u0026spr=https\\u0026sig=removed\"},\"storageAccountType\":\"Standard_ZRS\",\"targetRegions\":[{\"name\":\"West Europe\",\"regionalReplicaCount\":1,\"storageAccountType\":\"Standard_ZRS\"},{\"name\":\"East US\",\"regionalReplicaCount\":2,\"storageAccountType\":\"Standard_ZRS\"}]},\"safetyProfile\":{\"allowDeletionOfReplicatedLocations\":true}}}",
"id": "/subscriptions/subscription_id/resourceGroups/rg-opsstack-westeurope-001/providers/Microsoft.Compute/galleries/GAL_OPSSTACK_WESTEUROPE_001/applications/opsstack-agent-windows/versions/1.1.0",
"identity": [],
"ignore_body_changes": [
"properties.publishingProfile.source.media_link"
],
"ignore_casing": false,
"ignore_missing_property": true,
"location": "westeurope",
"locks": [
"/subscriptions/subscription_id/resourceGroups/rg-opsstack-westeurope-001/providers/Microsoft.Compute/galleries/GAL_OPSSTACK_WESTEUROPE_001/applications/opsstack-agent-windows"
],
"name": "1.1.0",
"output": "{\"properties\":{\"provisioningState\":\"Succeeded\"}}",
"parent_id": "/subscriptions/subscription_id/resourceGroups/rg-opsstack-westeurope-001/providers/Microsoft.Compute/galleries/GAL_OPSSTACK_WESTEUROPE_001/applications/opsstack-agent-windows",
"removing_special_chars": false,
"response_export_values": [
"properties.provisioningState"
],
"schema_validation_enabled": true,
"tags": {
"department": "opsstack",
"owner": "cloudeteer",
"provisioner": "Terraform"
},
"timeouts": null,
"type": "Microsoft.Compute/galleries/applications/versions@2022-03-03"
},
"sensitive_attributes": [],
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJkZWxldGUiOjE4MDAwMDAwMDAwMDAsInJlYWQiOjMwMDAwMDAwMDAwMCwidXBkYXRlIjoxODAwMDAwMDAwMDAwfX0=",
"dependencies": [
"module.opsstack.azurerm_gallery_application.windows",
"module.opsstack.azurerm_public_ip.this",
"module.opsstack.azurerm_resource_group.this",
"module.opsstack.azurerm_shared_image_gallery.this",
"module.opsstack.azurerm_storage_account.gallery",
"module.opsstack.azurerm_storage_blob.linux",
"module.opsstack.azurerm_storage_blob.windows",
"module.opsstack.azurerm_storage_container.gallery_opsstack_installer_scripts",
"module.opsstack.data.azurerm_resource_group.this",
"module.opsstack.data.azurerm_storage_account_blob_container_sas.this",
"module.opsstack.random_string.opsstack_gal_storage_account_name"
]
},
Differents to observe:
for_each
It's noticeable that my mediaLink contains a lot of special chars, maybe this causes the issue.
Thanks for providing this!
I think we need to inspect the state in v1.13.1, would you please help with this experiment?
Hi, in meanwhile, we upgrade to 0.14 together with OpenTofu 1.8 and the issue has been resolved without doing anything else.
After upgrading from 1.12 to 1.13.1, I have multiple plan issues: