`import` cannot evaluate resources which require state upgrades

casstait commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) 3.0.1

Affected Resource(s)

azurerm_mssql_database

Terraform Configuration Files

resource "azurerm_mssql_database" "mssql_database" { name = "tentaclearmy-mssql-database-${lower(var.environment)}" server_id = azurerm_mssql_server.mssql_server.id max_size_gb = 2 sku_name = "Basic" }

Expected Behaviour

terraform plan or import should succeed

Actual Behaviour

Receive the following error:

Error: Invalid resource instance data in state
│
│   on /Users/.../Octopus/TentacleArmy/terraform/main.tf line 71:
│   71: resource "azurerm_mssql_server" "mssql_server" {
│
│ Instance azurerm_mssql_server.mssql_server data could not be decoded from the state: unsupported attribute "extended_auditing_policy".

Steps to Reproduce

in main.tf

resource "azurerm_mssql_database" "mssql_database" {
  name             = "tentaclearmy-mssql-database-${lower(var.environment)}"
  server_id       = azurerm_mssql_server.mssql_server.id
  max_size_gb = 2
  sku_name      = "Basic"
}

terraform import azurerm_mssql_database.example /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group1/providers/Microsoft.Sql/servers/server1/databases/example1

Important Factoids

AzureRM recently updated a major version to 3.0

mbfrahry commented 2 years ago

Hey @casstait, I've been trying a number of ways to recreate this issue but haven't been able to. Could you share a bit more on how you got to the issue you're seeing? What version of Azure were you upgrading from and do you have a way to consistently reproduce this issue that you could share?

katbyte commented 2 years ago

Also if this continues to be an issue casstait a copy of the debug lo with TF_LOG=TRACE terraform import would be very helpful!

jbardin commented 2 years ago

Hi @casstait,

Thanks for filing the issue! This appears to be a result of the rather limited context within which import operates, which means it's not able to upgrade existing state to match a new provider version. The workaround here should be to complete an apply operation first to ensure the existing resources are refreshed and upgraded to the new provider version.

casstait commented 2 years ago

@jbardin The apply produced the same error. The workaround for us was to just manually manipulate the state until it was in the desired outcome.

Maybe the provider code could be a bit clever and notice what schema version a resource is using and then if the provider version doesn't match that it can update the state of a resource when these kind of things happen. I understand that's a big ask and out of scope for this issue but just a forwarding suggsetion.

jbardin commented 2 years ago

Hi @casstait,

Thanks, I would not expect the apply to produce the same error here, since state upgrades should be handled during the normal plan. I'm not sure yet if we have 2 different bugs, or somehow the behavior is related between the 2 operations. Would it be possible to provider the output of TF_LOG_CORE=trace from the apply failure?

Thanks!

casstait commented 2 years ago

@jbardin sorry can't get the trace logs for you. Like I said we manually manipulated the state until the apply worked. Wouldn't be able to get the trace logs unless I revert all my manual changes which I can't justify to do atm.

jbardin commented 2 years ago

Thanks for following up @casstait. In that case I'll keep this issue labeled as-is for now to see if we can replicate the import problem, and we can use the other issues for any plan or refresh errors.

Meandron commented 2 years ago

I have the same issue updating from azurerm 2.97.0 to 3.0.2 (with terraform plan).

I enabled trace log (TF_LOG=Trace and TF_LOG_CORE=Trace), however, as the TF handles quite many resources, I cannot copy it here. On first glance, I didn't notice anything helpful in the logs, just found:

2022-04-25T09:59:34.810Z [TRACE] provider.terraform-provider-azurerm_v3.0.2_x5: Received request: tf_resource_type=azurerm_mssql_server_extended_auditing_policy tf_proto_version=5 tf_provider_addr=provider tf_req_id=99c5236d-1522-c064-9573-75662ac711c4 tf_rpc=ValidateResourceTypeConfig @caller=/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:495 @module=sdk.proto timestamp=2022-04-25T09:59:34.809Z 2022-04-25T09:59:34.810Z [TRACE] provider.terraform-provider-azurerm_v3.0.2_x5: Calling downstream: tf_proto_version=5 tf_resource_type=azurerm_mssql_server_extended_auditing_policy tf_provider_addr=provider tf_req_id=99c5236d-1522-c064-9573-75662ac711c4 tf_rpc=ValidateResourceTypeConfig @caller=/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:502 @module=sdk.proto timestamp=2022-04-25T09:59:34.809Z 2022-04-25T09:59:34.810Z [TRACE] provider.terraform-provider-azurerm_v3.0.2_x5: Called downstream: @caller=/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:508 tf_proto_version=5 tf_rpc=ValidateResourceTypeConfig @module=sdk.proto tf_provider_addr=provider tf_req_id=99c5236d-1522-c064-9573-75662ac711c4 tf_resource_type=azurerm_mssql_server_extended_auditing_policy timestamp=2022-04-25T09:59:34.809Z 2022-04-25T09:59:34.810Z [TRACE] provider.terraform-provider-azurerm_v3.0.2_x5: Served request: tf_req_id=99c5236d-1522-c064-9573-75662ac711c4 tf_rpc=ValidateResourceTypeConfig @caller=/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:514 @module=sdk.proto tf_proto_version=5 tf_provider_addr=provider tf_resource_type=azurerm_mssql_server_extended_auditing_policy timestamp=2022-04-25T09:59:34.809Z 2022-04-25T09:59:34.810Z [TRACE] vertex "module.mssql_server.azurerm_mssql_server_extended_auditing_policy.sql_server_extended_auditing_policy": visit complete

And finally:

2022-04-25T09:59:44.913Z [TRACE] dag/walk: upstream of "module.mssql_server.azurerm_mssql_server_extended_auditing_policy.sql_server_extended_auditing_policy (expand)" errored, so skipping

In the state file, there is: { "module": "module.mssql_server", "mode": "managed", "type": "azurerm_mssql_server_extended_auditing_policy", "name": "sql_server_extended_auditing_policy", "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]", "instances": [ [skipped]

and:

"module": "module.mssql_server", "mode": "managed", "type": "azurerm_mssql_server", "name": "sql_server", "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]", "instances": [ { "schema_version": 0, [skipped...] "connection_policy": "Default", "extended_auditing_policy": [ { "log_monitoring_enabled": true, "retention_in_days": 91, "storage_account_access_key": "", "storage_account_access_key_is_secondary": false, "storage_account_subscription_id": "", "storage_endpoint": "<my_ep>" } ],

If there is anything else I could provide you with, let me know!

Meandron commented 2 years ago

Any news on this?

apparentlymart commented 2 years ago

Hi all!

Terraform v1.3 includes a rework of how Terraform Core implements importing so that it shares more code with the logic for generating a normal plan. That includes generating normal planning nodes (which includes the state upgrade behavior) for any resource instances that are not participating in the current import operation.

Therefore I believe this should be fixed in v1.3.0 and later. If you see similar behavior in those versions, please open a new issue for it since we will need to start fresh with new trace logs that align with the new implementation. The trace logs and other debugging we did here are now unfortunately invalidated by the import implementation being significantly different than it was in earlier versions of Terraform.

Thanks for reporting this!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform