[ISSUE] Issue with `databricks_storage_credential` resource for Azure Databricks

[kingnathanal]

Configuration

provider "databricks" {
  host = var.global_databricks_workspace_url
}

resource "databricks_storage_credential" "this" {
  name  = "a${var.asms_number}-${module.tagging.environment_label}-musea2-${var.scope}-sc-dbxac"
  owner = var.platform_ad_group_name
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.this.id
    managed_identity_id = azurerm_user_assigned_identity.databricks.id
  }
  comment = "storage cred managed identity credential managed by TF"
}

resource "databricks_grants" "external_creds" {
  storage_credential = databricks_storage_credential.this.id
  grant {
    principal  = var.privileged_ad_group_name
    privileges = ["ALL PRIVILEGES"]
  }
}

resource "databricks_cluster" "unity_shared" {
  cluster_name            = "a${var.asms_number}-${var.scope}-shared-cluster"
  spark_version           = data.databricks_spark_version.latest.id
  node_type_id            = data.databricks_node_type.this.id
  autotermination_minutes = 60
  autoscale {
    min_workers = 1
    max_workers = 2
  }
  azure_attributes {
    availability = "ON_DEMAND_AZURE"
  }
  data_security_mode = "USER_ISOLATION"
}

Expected Behavior

Should be able to create storage credential resource using Service Principal

Actual Behavior

Terraform fails during apply with an error:

│ Error: cannot create storage credential: Refresh token not found for userId: Some(586502210709458)
│ 
│   with databricks_storage_credential.this,
│   on main.tf line 263, in resource "databricks_storage_credential" "this":
│  263: resource "databricks_storage_credential" "this" {
│ 
╵

Terraform and provider versions

Terraform version 1.3.7
databricks = {
      source  = "databricks/databricks"
      version = "1.30.0"
    }

Is it a regression?

Have not tried this with any other version

Debug Output

databricks_storage_credential.this: Creating...
2023-12-12T02:41:03.548Z [DEBUG] databricks_storage_credential.this: applying the planned Create change
2023-12-12T02:41:03.550Z [DEBUG] provider.terraform-provider-databricks_v1.30.0: setting computed for "databricks_gcp_service_account" from ComputedKeys: timestamp=2023-12-12T02:41:03.550Z
2023-12-12T02:41:04.150Z [DEBUG] provider.terraform-provider-databricks_v1.30.0: non-retriable error: Refresh token not found for userId: Some(586502210709458): @module=databricks tf_req_id=94e43f13-cef1-ae67-5087-6e6419683cd6 tf_rpc=ApplyResourceChange tf_provider_addr=registry.terraform.io/databricks/databricks tf_resource_type=databricks_storage_credential @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 timestamp=2023-12-12T02:41:04.150Z
2023-12-12T02:41:04.151Z [DEBUG] provider.terraform-provider-databricks_v1.30.0: POST /api/2.1/unity-catalog/storage-credentials
> {
>   "azure_managed_identity": {
>     "access_connector_id": "/subscriptions/80d4a2fc-1764-4d22-819e-6d2109be33fb/resourceGroups/a219315-dt9-musea2-rg-ocrm/pr... (85 more bytes)",
>     "managed_identity_id": "/subscriptions/80d4a2fc-1764-4d22-819e-6d2109be33fb/resourceGroups/a219315-dt9-musea2-rg-ocrm/pr... (94 more bytes)"
>   },
>   "comment": "storage cred managed identity credential managed by TF",
>   "name": "a219315-dt9-musea2-ocrm-sc-dbxac"
> }
< HTTP/2.0 404 Not Found
< {
<   "details": [
<     {
<       "@type": "type.googleapis.com/google.rpc.RequestInfo",
<       "request_id": "0b7ab091-92ef-4b33-93df-ae25faf67616",
<       "serving_data": ""
<     }
<   ],
<   "error_code": "RESOURCE_DOES_NOT_EXIST",
<   "message": "Refresh token not found for userId: Some(586502210709458)"
< }: @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=94e43f13-cef1-ae67-5087-6e6419683cd6 @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_resource_type=databricks_storage_credential tf_rpc=ApplyResourceChange timestamp=2023-12-12T02:41:04.150Z
2023-12-12T02:41:04.151Z [ERROR] provider.terraform-provider-databricks_v1.30.0: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="cannot create storage credential: Refresh token not found for userId: Some(586502210709458)" tf_provider_addr=registry.terraform.io/databricks/databricks @module=sdk.proto diagnostic_detail= tf_proto_version=5.4 tf_req_id=94e43f13-cef1-ae67-5087-6e6419683cd6 tf_resource_type=databricks_storage_credential @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 tf_rpc=ApplyResourceChange timestamp=2023-12-12T02:41:04.151Z
2023-12-12T02:41:04.152Z [ERROR] vertex "databricks_storage_credential.this" error: cannot create storage credential: Refresh token not found for userId: Some(586502210709458)
2023-12-12T02:41:04.319Z [INFO]  Starting apply for module.domain_analytics_nsg.azurerm_network_security_group.this
module.domain_analytics_nsg.azurerm_network_security_group.this:

Important Factoids

My Terraform is normally applied using a Service Principal, but when I change the Databricks provider to use a PAT like below:

provider "databricks" {
  host = var.global_databricks_workspace_url
  token     = "################################"
  auth_type = "pat"
}

I am able to create the storage credential no issues.

I am also able to create other Databricks resources like the Compute Cluster with just the Service Principal also, shown in the configuration above.

The error makes this hard to know if this is a permission issue or an issue with the provider

[hargut]

Seeing the same error with databricks cli and a service principal in use on an Azure env.

[kingnathanal]

@hargut Thanks I dont feel crazy anymore. Did this work for you before? Thinking of rolling back to a earlier provider version just to see if it works but it its a little shooting darts to know which version to use.

[hargut]

@kingnathanal Sorry, I've no idea about that as I've not used it like that before. I've also opened a ticket on the cli repo.

Creation with a user token as well works fine with the cli, so it seems to be specific for the service principal.

[mgyucht]

This is likely an issue with the backing service. I'll raise this issue to the underlying team to see if they can take a look.

[hargut]

Thanks for having a closer look. :+1:

[kingnathanal]

@mgyucht Any updates on this? Thanks

[hargut]

https://github.com/databricks/cli/issues/1108#issuecomment-1905946370

[wschultz-boxboat]

Receiving the same error with Terraform 1.6.6 and databricks/databricks 1.36.1

[wschultz-boxboat]

Given the inclusion of a data resource for storage_credential being added to 1.37.0, I decided to give azurerm_storage_credential another try on 1.37.0, and it still fails with the same error. Terraform 1.6.6

[nkvuong]

@kingnathanal @wschultz-boxboat just want to check if this is the same issue you're facing - https://github.com/databricks/terraform-provider-databricks/issues/2828#issuecomment-1966424366

[wschultz-boxboat]

@kingnathanal @wschultz-boxboat just want to check if this is the same issue you're facing - #2828 (comment)

@nkvuong it works if the Service Principal is granted Account Admin and the storage credential resource uses an account scoped provider config.

This won't work for my use case though as this service principal operates solely in one Databricks workspace and should not have Account Admin level of permissions.

Is there work on the roadmap to get the Service Principal's permissions to least privilege?

[antsok]

Are there updates on the fix for this issue ?

[GavWall]

I can confirm the issue still occurs with version 1.53.0 of the databricks terraform provider, with the workaround of requiring the Managed Identity to be granted Account Admin rights (which is exactly what we'd like to avoid - the MI/SP was already a metastore owner, which should have been enough in our scenario).

[alexott]

@GavWall @antsok it's an UC backend restriction, not the Terraform one - open the support ticket against UC API to solve it.

[antsok]

@alexott I wonder if the support will want to look at the ticket if it is created since the case is documented as unsupported. https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/storage-credentials

[alexott]

My point is that it's not a terraform issue...