[ISSUE] Resource `databricks_permission_assignment` can't find groups by ID

[camilo-s]

I'm unable to enable account-level groups at the workspace level in our Azure Databricks deployment.

Context:

• I've configured SCIM provisioning using Azure AD according to the documentation.
• I've assigned groups to the Databricks' Enterprise Application, which I can successfully see in the Databricks Account console after sync.
• I need to enable these groups in certain workspaces, and then assign permissions/grants/entitlements at the workspace level via Terraform.
• I'm using two instances of the Databricks provider: once for Account-level resources, to read the groups, and once for Workspace-level resources.
• The databricks_group data source is able to successfully read the groups at the account level.
• When referring these groups by ID for permission assignment at the workspace level, the Databricks provider fails claiming it's not able to find the group IDs.

Configuration

# root module

terraform {
  backend "azurerm" {
    use_azuread_auth = true
  }
  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.9.0"
    }
  }
}

provider "databricks" {
  host                        = module.databricks_workspace.databricks_workspace_url
  azure_workspace_resource_id = module.databricks_workspace.azure_workspace_resource_id
  azure_use_msi               = true
}

provider "databricks" {
  host          = "https://accounts.azuredatabricks.net"
  account_id    = var.databricks_account_id
  azure_use_msi = true

  alias = "account"
}

module "databricks_resources" {
  source = "../../modules/databricks-resources"

  metastore_id = var.metastore_id
  workspace_id = var.databricks_workspace_id
  azure_ad_groups = {
    group1 = {
      databricks = {
        workspace_access      = false
        databricks_sql_access = false
      }
    }
  }

  depends_on = [
    module.databricks_workspace
  ]

  providers = {
    databricks.account = databricks.account
  }
}

# Module databricks-resources.tf

terraform {
  required_providers {
    databricks = {
      configuration_aliases = [databricks.account]
      source                = "databricks/databricks"
    }
  }
}

resource "databricks_metastore_assignment" "this" {

  metastore_id = var.metastore_id
  workspace_id = var.databricks_workspace_id
}

data "databricks_group" "users" {
  for_each = var.azure_ad_groups

  display_name = each.key

  provider = databricks.account
}

resource "databricks_permission_assignment" "users" {
  for_each = var.azure_ad_groups

  principal_id = data.databricks_group.users[each.key].id
  permissions  = ["USERS"]
}

resource "databricks_entitlements" "users" {
  for_each = { for k, v in var.azure_ad_groups : k => v if v.databricks != null }

  group_id                   = data.databricks_group.users[each.key].id
  workspace_access           = each.value.databricks.workspace_access
  databricks_sql_access      = each.value.databricks.databricks_sql_access

  depends_on = [
    databricks_permission_assignment.users
  ]
}

Expected Behavior

After reading the groups in the data source, the databricks_permission_assignment gets deployed, enabling the groups in the workspace for further workspace-level operations on them.

Actual Behavior

terraform apply fails while creating the databricks_permission_assignment resource:

Error: cannot read permission assignment: 000000000000 not found
│ 
│   with module.databricks_resources.databricks_permission_assignment.users["group1"],
│   on ../../modules/databricks-resources/main.tf line 134, in resource "databricks_permission_assignment" "users":
│  134: resource "databricks_permission_assignment" "users" {

Steps to Reproduce

• Terraform version 1.4.5
• Databricks provider version 1.9.2 (I'm stuck at this version due to an issue regarding provider authentication starting from 1.10)

Debug Output

https://gist.github.com/camilo-s/f1de9bf2cff1853ec80dd9fe04d77f78

Important Factoids

• Terraform authenticates against Azure on behalf of a Service Principal using MSI. The SP has Contributor rights at the workspace level, and admin role at the Databricks account.
• Terraform runs on Ubuntu 20_04-lts-gen2

[camilo-s]

To be able to speak the permission assignment API at the account level, I tried using databricks_mws_permission_assignment instead.

The Provider is able to make the API call, which returns a 200, but fails afterwards when it tries to find the new permission among the existing ones:

2023-04-25T12:14:33.7178926Z 2023-04-25T12:14:33.346Z [INFO]  Starting apply for module.databricks_resources.databricks_mws_permission_assignment.users["group1"]
2023-04-25T12:14:33.7179448Z 2023-04-25T12:14:33.348Z [DEBUG] module.databricks_resources.databricks_mws_permission_assignment.users["group1"]: applying the planned Create change
2023-04-25T12:14:34.4944182Z 2023-04-25T12:14:34.493Z [DEBUG] provider.terraform-provider-databricks_v1.14.3: PUT /api/2.0/preview/accounts/***/workspaces/7181879217578353/permissionassignments/principals/784617661451777
2023-04-25T12:14:34.4944622Z > {
2023-04-25T12:14:34.4944744Z >   "permissions": [
2023-04-25T12:14:34.4945004Z >     "USERS"
2023-04-25T12:14:34.4945143Z >   ]
2023-04-25T12:14:34.4945250Z > }
2023-04-25T12:14:34.4945385Z < HTTP/2.0 200 OK
2023-04-25T12:14:34.4945503Z < {
2023-04-25T12:14:34.4945635Z <   "permissions": [
2023-04-25T12:14:34.4945773Z <     "USER"
2023-04-25T12:14:34.4945885Z <   ],
2023-04-25T12:14:34.4946016Z <   "principal": {
2023-04-25T12:14:34.4946173Z <     "display_name": "group1",
2023-04-25T12:14:34.4946369Z <     "group_name": "group1",
2023-04-25T12:14:34.4946534Z <     "principal_id": 784617661451777
2023-04-25T12:14:34.4946679Z <   }
2023-04-25T12:14:34.4946934Z < }: timestamp=2023-04-25T12:14:34.493Z
2023-04-25T12:14:35.0717290Z 2023-04-25T12:14:35.069Z [DEBUG] provider.terraform-provider-databricks_v1.14.3: GET /api/2.0/preview/accounts/***/workspaces/7181879217578353/permissionassignments
2023-04-25T12:14:35.0717810Z < HTTP/2.0 200 OK
2023-04-25T12:14:35.0719324Z < {
2023-04-25T12:14:35.0719662Z <   "permission_assignments": [
2023-04-25T12:14:35.0720521Z <     {
2023-04-25T12:14:35.0721791Z <       "permissions": [
2023-04-25T12:14:35.0722534Z <         "ADMIN"
2023-04-25T12:14:35.0722708Z <       ],
2023-04-25T12:14:35.0723365Z <       "principal": {
2023-04-25T12:14:35.0724277Z <         "display_name": "####",
2023-04-25T12:14:35.0724824Z <         "service_principal_name": "####",
2023-04-25T12:14:35.0725871Z <         "service_principal_name": "####"
2023-04-25T12:14:35.0726154Z <       }
2023-04-25T12:14:35.0726305Z <     },
2023-04-25T12:14:35.0726416Z <     {
2023-04-25T12:14:35.0726923Z <       "permissions": [
2023-04-25T12:14:35.0727636Z <         "ADMIN"
2023-04-25T12:14:35.0727803Z <       ],
2023-04-25T12:14:35.0728454Z <       "principal": {
2023-04-25T12:14:35.0729200Z <         "display_name": "####",
2023-04-25T12:14:35.0729940Z <         "service_principal_name": "####",
2023-04-25T12:14:35.0730830Z <         "user_name": "####"
2023-04-25T12:14:35.0731349Z <       }
2023-04-25T12:14:35.0732122Z <     },
2023-04-25T12:14:35.0732265Z <     {
2023-04-25T12:14:35.0732916Z <       "permissions": [
2023-04-25T12:14:35.0733095Z <         "ADMIN"
2023-04-25T12:14:35.0733699Z <       ],
2023-04-25T12:14:35.0733873Z <       "principal": {
2023-04-25T12:14:35.0734660Z <         "display_name": "####",
2023-04-25T12:14:35.0735156Z <         "service_principal_name": "####",
2023-04-25T12:14:35.0740851Z <         "service_principal_name": "####"
2023-04-25T12:14:35.0741189Z <       }
2023-04-25T12:14:35.0741322Z <     },
2023-04-25T12:14:35.0741990Z <     {
2023-04-25T12:14:35.0742297Z <       "permissions": [
2023-04-25T12:14:35.0742441Z <         "USER"
2023-04-25T12:14:35.0742571Z <       ],
2023-04-25T12:14:35.0742688Z <       "principal": {
2023-04-25T12:14:35.0742961Z <         "display_name": "####",
2023-04-25T12:14:35.0743726Z <         "service_principal_name": "####",
2023-04-25T12:14:35.0744675Z <         "service_principal_name": "####"
2023-04-25T12:14:35.0745222Z <       }
2023-04-25T12:14:35.0745362Z <     },
2023-04-25T12:14:35.0745999Z <     {
2023-04-25T12:14:35.0746172Z <       "permissions": [
2023-04-25T12:14:35.0746779Z <         "ADMIN"
2023-04-25T12:14:35.0746944Z <       ],
2023-04-25T12:14:35.0749024Z <       "principal": {
2023-04-25T12:14:35.0749729Z <         "display_name": "####",
2023-04-25T12:14:35.0750523Z <         "service_principal_name": "####",
2023-04-25T12:14:35.0751372Z <         "user_name": "####"
2023-04-25T12:14:35.0751883Z <       }
2023-04-25T12:14:35.0752036Z <     },
2023-04-25T12:14:35.0752722Z <     {
2023-04-25T12:14:35.0752876Z <       "permissions": [
2023-04-25T12:14:35.0753428Z <         "ADMIN"
2023-04-25T12:14:35.0753554Z <       ],
2023-04-25T12:14:35.0754147Z <       "principal": {
2023-04-25T12:14:35.0754801Z <         "display_name": "####",
2023-04-25T12:14:35.0755477Z <         "service_principal_name": "####",
2023-04-25T12:14:35.0756247Z <         "user_name": "####"
2023-04-25T12:14:35.0756979Z <       }
2023-04-25T12:14:35.0757222Z <     },
2023-04-25T12:14:35.0757356Z <     "... (9 additional elements)"
2023-04-25T12:14:35.0757671Z <   ]
2023-04-25T12:14:35.0757928Z < }: timestamp=2023-04-25T12:14:35.068Z
2023-04-25T12:14:35.0759712Z 2023-04-25T12:14:35.069Z [ERROR] provider.terraform-provider-databricks_v1.14.3: Response contains error diagnostic: tf_resource_type=databricks_mws_permission_assignment @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:55 diagnostic_summary="cannot read mws permission assignment: 784617661451777 not found" tf_proto_version=5.3 tf_req_id=650d248e-2f47-c037-97b0-fe21ab10b24b @module=sdk.proto diagnostic_detail= diagnostic_severity=ERROR tf_provider_addr=registry.terraform.io/databricks/databricks tf_rpc=ApplyResourceChange timestamp=2023-04-25T12:14:35.069Z
2023-04-25T12:14:35.0761005Z 2023-04-25T12:14:35.070Z [ERROR] vertex "module.databricks_resources.databricks_mws_permission_assignment.users[\"group1\"]" error: cannot read mws permission assignment: 784617661451777 not found

[philippbussche]

I came across the same error but it turns out that my permission was wrong. The permission would be "USER" and not "USERS". You also have this incorrect in your configuration @camilo-s . However the error message is also misleading.