[ISSUE] Provider from 1.24 enforces id check when importing `databricks_metastore_assignment` resource

[VOVELEE]

Configuration

resource "databricks_metastore_assignment" "this" {
  provider = databricks.workspace

  metastore_id = var.metastore_id
  workspace_id = var.workspace_id

  default_catalog_name = var.default_catalog_name
}

Expected Behavior

• The metastore assignment to be refreshed and changes applied if needed

Actual Behavior

• This piece of code worked correctly in databricks provider 1.23 and below.
• It looks like something has changed since 1.24 and when upgrade is done on the provider - planning of already created resources fails. It worth mentioning this one fails only on assignments created long time ago. Newly created assignments are refreshed correctly.
• Observation:
  • it looks like old metastore assignment resources get id with metastore_id/workspace_id
  • new metastore assignment resources get id with workspace_id|metastore_id

module.solutions.databricks_metastore_assignment.this: Refreshing state... [id=4f459f4c-efce-4af1-a511-f83e6c18cba2/4716493111111111]
module.solutions.databricks_metastore_assignment.dbw["new-one"]: Refreshing state... [id=1104224111111111|4f459f4c-efce-4af1-a511-f83e6c18cba2]

Steps to Reproduce

• terraform init --upgrade
• terraform plan

Terraform and provider versions

• Works in databricks provider 1.23 and below
• Error in 1.24 and above

Debug Output

Important Factoids

[patrickwilliamconway]

Bumping this - I'm also seeing issues with this resource.

Here is my resource configuration:

// https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_assignment
resource "databricks_metastore_assignment" "default_metastore" {
  count = var.enable_unity_catalog ? 1 : 0
  workspace_id         = local.workspace_id
  metastore_id         =  "omitted string value" 
  default_catalog_name = "hive_metastore"
}

and here is my terraform & provider versions:

> terraform version
Terraform v1.1.4
on darwin_arm64
+ provider registry.terraform.io/databricks/databricks v1.23.0

When I apply, the resource is nowhere to be found. Here are debug logs that reference my default_metastore.

> cat /tmp/tf-dbr-apply-1697035378.log | grep default_metastore

2023-10-11T10:43:02.566-0400 [DEBUG] ProviderTransformer: "module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore" (*terraform.NodeValidatableResource) needs module.main.provider["registry.terraform.io/databricks/databricks"].query_service
2023-10-11T10:43:02.584-0400 [DEBUG] ReferenceTransformer: "module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore" references: [module.main.module.query_service_account_level.var.enable_unity_catalog (expand) module.main.module.query_service_account_level.local.workspace_id (expand)]
2023-10-11T10:43:02.586-0400 [DEBUG] ReferenceTransformer: "module.main.module.query_service_account_level.databricks_mws_permission_assignment.add_company_group" references: [module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore module.main.module.query_service_account_level.databricks_group.company_group_uc module.main.module.query_service_account_level.local.workspace_id (expand)]
2023-10-11T10:43:03.170-0400 [DEBUG] ProviderTransformer: "module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore (expand)" (*terraform.nodeExpandPlannableResource) needs module.main.provider["registry.terraform.io/databricks/databricks"].query_service
2023-10-11T10:43:03.187-0400 [DEBUG] ReferenceTransformer: "module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore (expand)" references: [module.main.module.query_service_account_level.var.enable_unity_catalog (expand) module.main.module.query_service_account_level.local.workspace_id (expand)]
2023-10-11T10:43:03.192-0400 [DEBUG] ReferenceTransformer: "module.main.module.query_service_account_level.databricks_mws_permission_assignment.add_company_group (expand)" references: [module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore (expand) module.main.module.query_service_account_level.databricks_group.company_group_uc (expand) module.main.module.query_service_account_level.local.workspace_id (expand)]
2023-10-11T10:43:06.395-0400 [INFO]  ReferenceTransformer: reference not found: "databricks_metastore_assignment.default_metastore"
2023-10-11T10:43:07.278-0400 [DEBUG] ProviderTransformer: "module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore (expand)" (*terraform.nodeExpandApplyableResource) needs module.main.provider["registry.terraform.io/databricks/databricks"].query_service
2023-10-11T10:43:07.298-0400 [INFO]  ReferenceTransformer: reference not found: "databricks_metastore_assignment.default_metastore#destroy"
2023-10-11T10:43:07.298-0400 [DEBUG] ReferenceTransformer: "module.main.module.query_service_account_level[0].databricks_mws_permission_assignment.add_company_group" references: [module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore (expand) module.main.module.query_service_account_level.databricks_group.company_group_uc (expand) module.main.module.query_service_account_level[0].databricks_group.company_group_uc module.main.module.query_service_account_level[0].databricks_group.company_group_uc module.main.module.query_service_account_level.local.workspace_id (expand)]
2023-10-11T10:43:07.302-0400 [DEBUG] ReferenceTransformer: "module.main.module.query_service_account_level.databricks_metastore_assignment.default_metastore (expand)" references: [module.main.module.query_service_account_level.var.enable_unity_catalog (expand)]
      "name": "default_metastore",

Seems like this log line is most relevant?

2023-10-11T10:43:06.395-0400 [INFO]  ReferenceTransformer: reference not found: "databricks_metastore_assignment.default_metastore"

I get with databricks version 1.20.0, 1.23.0, 1.27.0.

I'm also unable to import it into state:

1. since it doesn't exist from TF perspective:
2. error with shell

> terraform import databricks_metastore_assignment.default_metastore "myworkspaceid|metastoreid"
/bin/sh: metastoreid: command not found
Error: resource address "databricks_metastore_assignment.default_metastore" does not exist in the configuration.

Before importing this resource, please create its configuration in the root module. For example:

resource "databricks_metastore_assignment" "default_metastore" {
  # (resource arguments)
}

replaing | with / in import like OP suggested yields same result, just without shell error.

[patrickwilliamconway]

hey @nkvuong, tagging you to get your attention on this (you're the most visible Databricks name I see 😄).

This has been pretty disruptive - my team is trying to deploy a lot of workspaces and this doesn't seem to be working.

[nkvuong]

@patrickwilliamconway we missed a state upgrader when the id is changed - I'll look into this issue this week

[patrickwilliamconway]

thank you 🙏

[VOVELEE]

thanks looking forward to test it :)

[nkvuong]

@VOVELEE I just looked through the source code, and the id for databricks_metastore_assignment has always been workspace_id|metastore_id since the beginning. So it is a bit puzzling that you are seeing metastore_id/workspace_id as the id in the state. This makes us quite hesitant in merging the state upgrader.

I re-tested with provider 1.23, and the ids are created in the workspace_id|metastore_id format as well. Could you confirm?

[VOVELEE]

@nkvuong I can confirm that for new metastore assignments it is workspace_id|metastore_id I just checked the history of this particular faulty resource. It looks like it has been imported in the TF State with provider 1.8.0 because it has been manually created in the past. Since then this resource has been always correctly recognized by the provider. However in version 1.24 and above it started failing.

[nkvuong]

@VOVELEE I think I know the issue. Basically the read function of databricks_metastore_assignment does not work properly with the workspace-level API. This code does not use the stored id in the state, and only sets the metastore_id, and therefore would accept any id

➜  unity terraform import databricks_metastore_assignment.this thisiscompletelyrubbish                             
databricks_metastore_assignment.this: Importing from ID "thisiscompletelyrubbish"...
databricks_metastore_assignment.this: Import prepared!
  Prepared databricks_metastore_assignment for import
databricks_metastore_assignment.this: Refreshing state... [id=thisiscompletelyrubbish]

Import successful!

In 1.24.0 we switch to Go SDK, which would fail the read as it actually parses the id properly.

[nkvuong]

discussed this with @mgyucht, we could relax the id check, but fundamentally databricks_metastore_assignment resource needs to be managed at account-level, see comment here

I would recommend to re-import the resource using account-level provider to ensure correct usage going forward

[VOVELEE]

thanks @nkvuong Will try on Monday and provide feedback

[VOVELEE]

closing this because cannot reproduce it anymore