[ISSUE] Issue with `databricks_grants` resource

[OS-ahmedfouzan]

Configuration

resource "databricks_grants" "metastore" {
  provider  = databricks.mws
  count     = var.databricks_create_metastore ? 1 : 0
  metastore = databricks_metastore.this[0].id
  grant {
    principal  = databricks_group.admin_group[0].display_name
    privileges = ["CREATE_EXTERNAL_LOCATION"]
  }
  depends_on = [databricks_metastore.this[0]]
}

Expected Behavior

Should grant the admin group permissions to create external location on the metastore

Actual Behavior

Throwing error

│ Error: cannot create grants: invalid character '<' looking for beginning of value
│ 
│   with module.databricks_metastore.databricks_grants.metastore[0],
│   on metastore/access.tf line 19, in resource "databricks_grants" "metastore":
│   19: resource "databricks_grants" "metastore" {

Steps to Reproduce

1. Create metastore and admin group. Use the same to create the grant
2. Run terraform apply

Terraform and provider versions

1.23.0

Debug Output

2023-08-18T23:57:50.575+0530 [DEBUG] module.databricks_workspace_resources.databricks_grants.datalake: applying the planned Update change
2023-08-18T23:57:53.305+0530 [ERROR] provider.terraform-provider-databricks_v1.23.0: Response contains error diagnostic: tf_resource_type=databricks_grants @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="cannot create grants: invalid character '<' looking for beginning of value" tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=81b59909-2886-e262-6d8e-ba9c03258ab3 @module=sdk.proto tf_proto_version=5.3 tf_rpc=ApplyResourceChange timestamp=2023-08-18T23:57:53.305+0530
2023-08-18T23:57:53.307+0530 [DEBUG] State storage *remote.State declined to persist a state snapshot
2023-08-18T23:57:53.307+0530 [ERROR] vertex "module.databricks_metastore.databricks_grants.metastore[0]" error: cannot create grants: invalid character '<' looking for beginning of value
module.databricks_workspace_resources.databricks_grants.datalake_credential: Modifications complete after 3s [id=storage_credential/<credential-name>]
2023-08-18T23:57:53.750+0530 [DEBUG] State storage *remote.State declined to persist a state snapshot
module.databricks_workspace_resources.databricks_grants.datalake: Modifications complete after 3s [id=external_location/<external-location>]
2023-08-18T23:57:54.041+0530 [DEBUG] State storage *remote.State declined to persist a state snapshot
2023-08-18T23:57:54.044+0530 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-08-18T23:57:54.047+0530 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.23.0/darwin_arm64/terraform-provider-databricks_v1.23.0 pid=44387
2023-08-18T23:57:54.047+0530 [DEBUG] provider: plugin exited
2023-08-18T23:57:54.047+0530 [DEBUG] states/remote: state read serial is: 34; serial is: 34
2023-08-18T23:57:54.047+0530 [DEBUG] states/remote: state read lineage is: 261aa823-a82a-cb5b-a19b-f89bf95efdcd; lineage is: 261aa823-a82a-cb5b-a19b-f89bf95efdcd

Error: cannot create grants: invalid character '<' looking for beginning of value

  with module.databricks_metastore.databricks_grants.metastore[0],
  on metastore/access.tf line 19, in resource "databricks_grants" "metastore":
  19: resource "databricks_grants" "metastore" {

Important Factoids

[rondefreitas]

This appears related to https://github.com/databricks/terraform-provider-databricks/pull/2182 which they're already working on

In the meantime, @OS-ahmedfouzan , you'll need to perform this action from a workspace login context, as per the documentation:

Unity Catalog APIs are accessible via workspace-level APIs. This design may change in the future. Account-level principal grants can be assigned with any valid workspace as the Unity Catalog is decoupled from specific workspaces. More information in the official documentation.

[nkvuong]

@rondefreitas just a correction - account-level APIs will only be supported for a few UC resources, namely metastore & storage credentials