[ISSUE] Issue with `databricks_token`, `databricks_secret_scope` and `databricks_workspace_conf` resource

[russellseymour]

Configuration

resource "databricks_token" "pat" {
  comment = var.databricks_pat_comment
  // 120 day token
  lifetime_seconds = 120 * 24 * 60 * 60
  depends_on       = [module.adb]
}

resource "azurerm_key_vault_secret" "databricks_token" {
  name         = var.databricks-token
  value        = databricks_token.pat.token_value
  key_vault_id = module.kv_default.id
  depends_on   = [module.adb, module.kv_default]
}

resource "databricks_secret_scope" "kv" {
  name = var.databricks_secret_scope_kv

  keyvault_metadata {
    resource_id = module.kv_default.id
    dns_name    = module.kv_default.vault_uri
  }
  depends_on = [module.adb, module.kv_default]
}

The databricks module that we have created looks like:

resource "azurerm_databricks_workspace" "example" {
  name                                  = var.resource_namer
  location                              = var.resource_group_location
  resource_group_name                   = var.resource_group_name
  sku                                   = var.databricks_sku
  public_network_access_enabled         = var.public_network_access_enabled
  network_security_group_rules_required = var.managed_vnet ? null : var.network_security_group_rules_required
  managed_resource_group_name           = "databricks-rg-${var.resource_group_name}"
  load_balancer_backend_address_pool_id = var.create_lb ? azurerm_lb_backend_address_pool.lb_be_pool[0].id : null

  dynamic "custom_parameters" {
    for_each = var.enable_private_network == false ? toset([]) : toset([1])
    content {
      no_public_ip                                         = true
      public_subnet_name                                   = var.managed_vnet ? null : (var.create_subnets ? azurerm_subnet.public_subnet[0].name : var.public_subnet_name)
      private_subnet_name                                  = var.managed_vnet ? null : (var.create_subnets ? azurerm_subnet.private_subnet[0].name : var.private_subnet_name)
      virtual_network_id                                   = var.managed_vnet ? null : var.virtual_network_id
      vnet_address_prefix                                  = var.managed_vnet ? null : (var.vnet_address_prefix == "" ? null : var.vnet_address_prefix)
      public_subnet_network_security_group_association_id  = var.managed_vnet ? null : azurerm_subnet_network_security_group_association.public[0].id
      private_subnet_network_security_group_association_id = var.managed_vnet ? null : azurerm_subnet_network_security_group_association.private[0].id
      nat_gateway_name                                     = var.managed_vnet ? null : (var.create_nat ? azurerm_nat_gateway.nat[0].name : null)
      public_ip_name                                       = var.managed_vnet ? null : (var.create_nat ? azurerm_public_ip.pip[0].name : null)
    }
  }

  tags = var.resource_tags
  lifecycle {
    ignore_changes = [
      tags,
    ]
  }
  depends_on = [azurerm_subnet.public_subnet, azurerm_subnet.private_subnet]
}

Expected Behavior

When running terraform destroy the resources should be removed from Azure

Actual Behavior

Although the provider was able to create the resources using a Service Principal with Contributor access it is not able to delete them. We get the following error messages:

Error: cannot delete secret scope: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: azure_client_secret=***, azure_client_id=d474405c-xxxx-4c99-xxxx-da12502b13b4, azure_tenant_id=d483ed84-f7bf-xxxx-xxxx-fb250feccf8f. Env: ARM_CLIENT_SECRET, ARM_CLIENT_ID, ARM_TENANT_ID

Error: cannot delete token: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: azure_client_secret=***, azure_client_id=d474405c-xxxx-4c99-xxxx-da12502b13b4, azure_tenant_id=d483ed84-f7bf-xxxx-xxxx-fb250feccf8f. Env: ARM_CLIENT_SECRET, ARM_CLIENT_ID, ARM_TENANT_ID

Error: cannot delete workspace conf: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: azure_client_secret=***, azure_client_id=d474405c-xxxx-4c99-xxxx-da12502b13b4, azure_tenant_id=d483ed84-f7bf-xxxx-xxxx-fb250feccf8f. Env: ARM_CLIENT_SECRET, ARM_CLIENT_ID, ARM_TENANT_ID

Steps to Reproduce

Create an Azure Data Bricks instance in Azure using Terraform and create the toke, secret_scope and workspace_conf resources, then run

1. terraform plan -out tfplan
2. terraform apply tfplan
3. terraform plan -destroy -out destroy.tfplan
4. terraform destroy destroy.tfplan

Terraform and provider versions

Terraform v1.5.1
on linux_amd64

Is it a regression?

Not that I am aware of

Debug Output

https://gist.github.com/russellseymour/cfd4df94f6fc5a0300bd7337528fa6fb

Important Factoids

No

Would you like to implement a fix?

No as I do not knoww what the issue is

[alexott]

show the provider block...

[stonenw-va]

Not sure about the OP, but here is my provider block. I have the same issue when using version 1.30.0. The reason I am using an older version is due to other issues regarding USGovernment. I think this is a sequencing error internal to Terraform databricks.

provider "databricks" { alias = "databricks_workspace1" host = length(module.databricks) > 0 ? module.databricks[local.databricks_keys[0]].databricks_workspace_url : "empty" azure_workspace_resource_id = length(module.databricks) > 0 ? module.databricks[local.databricks_keys[0]].databricks_workspace.id : "empty" azure_environment = "usgovernment" }

For additional context I have two modules, one which creates the Databricks Workspace in Azure, and then another which configures the workspace using the values from the Databricks Workspace and configuring the provider above.

[russellseymour]

We have written a module that creates an Azure Databricks Workspace. The output of this module is the host and the azure_workspace_resource_id which we reference in the provider:

provider "databricks" {
  host                        = module.adb.databricks_hosturl
  azure_workspace_resource_id = module.adb.adb_databricks_id
  auth_type                   = "azure-client-secret"
}

However we had a thought that when being destroyed the module information is being removed before we get to these resources, so the host and azure_workspace_resource_id is no longer valid. So as a test we changed it to this:

provider "databricks" {
  host                        = module.adb.databricks_hosturl != "" ? module.adb.databricks_hosturl : var.adb_databricks_hosturl
  azure_workspace_resource_id = module.adb.adb_databricks_id != "" ? module.adb.adb_databricks_id : var.adb_databricks_id
  auth_type                   = "azure-client-secret"
}

Now when we run the destroy we set the variables accordingly so they do not come from the module output anymore. This has changed the error we get to:

infra:destroy:plan: │ Error: cannot read token: Invalid resource ID
infra:destroy:plan: │
infra:destroy:plan: │   with databricks_token.pat,
infra:destroy:plan: │   on main.tf line 359, in resource "databricks_token" "pat":
infra:destroy:plan: │  359: resource "databricks_token" "pat" {
infra:destroy:plan: │
infra:destroy:plan: ╵
infra:destroy:plan: ╷
infra:destroy:plan: │ Error: cannot read secret scope: Invalid resource ID
infra:destroy:plan: │
infra:destroy:plan: │   with databricks_secret_scope.kv,
infra:destroy:plan: │   on main.tf line 381, in resource "databricks_secret_scope" "kv":
infra:destroy:plan: │  381: resource "databricks_secret_scope" "kv" {
infra:destroy:plan: │
infra:destroy:plan: ╵
infra:destroy:plan: ╷
infra:destroy:plan: │ Error: cannot read workspace conf: Invalid resource ID
infra:destroy:plan: │
infra:destroy:plan: │   with databricks_workspace_conf.this[0],
infra:destroy:plan: │   on main.tf line 391, in resource "databricks_workspace_conf" "this":
infra:destroy:plan: │  391: resource "databricks_workspace_conf" "this" {

There does not appear to be a way to provide additional properties that may make this work.

Is it possible to actually destroy Databricks resources using Terraform?