[ISSUE] Account-level auth is broken: provider is not picking up account-level OAuth token from Databricks CLI

[nfx]

It's not possible to fully automate a fresh Databricks Account in one apply using Databricks CLI authentication.

Expected Behavior

After running databricks auth login --host https://accounts.cloud.databricks.com --account-id XXXX, I expect the workspace provider to pick up the token it used to authenticate to account console.

provider "databricks" {
  alias      = "account"
  host       = module.defaults.aws_account_console
  account_id = module.defaults.aws_databricks_account_id
}

...

resource "databricks_mws_workspaces" "this" {
  provider       = databricks.account
  account_id     = module.defaults.aws_databricks_account_id
  aws_region     = module.defaults.aws_region
  workspace_name = "${local.prefix}-ucws"

  credentials_id           = databricks_mws_credentials.this.credentials_id
  storage_configuration_id = databricks_mws_storage_configurations.this.storage_configuration_id
}

...

provider "databricks" {
  alias = "workspace"
  host  = databricks_mws_workspaces.this.workspace_url
}

Actual Behavior

I received cannot configure default credentials. Config: host=<host from databricks_mws_workspaces.this.workspace_url>.

Attempt 2

SPN auth doesn't work as well:

resource "databricks_service_principal" "owner" {
  provider     = databricks.account
  display_name = "owner-sp"
}

resource "databricks_service_principal_role" "account_admin" {
  provider             = databricks.account
  service_principal_id = databricks_service_principal.owner.id
  role                 = "account_admin"
}

resource "databricks_service_principal_secret" "ower" {
  provider             = databricks.account
  service_principal_id = databricks_service_principal.owner.id
}

...

provider "databricks" {
  alias         = "workspace"
  host          = databricks_mws_workspaces.this.workspace_url
  client_id     = databricks_service_principal.owner.application_id
  client_secret = databricks_service_principal_secret.ower.secret
}

results in │ Error: inner token: oauth2: "invalid_client" "Client authentication failed"

Attempt 3

Assigning workspace access permission doesn't work either: │ Error: cannot create mws permission assignment: Permission assignment APIs are not available for this workspace., until the UC is enabled. and UC requires a workspace to execute APIs.

resource "databricks_mws_permission_assignment" "owner" {
  provider     = databricks.account
  principal_id = databricks_service_principal.owner.id
  workspace_id = databricks_mws_workspaces.this.workspace_id
  permissions  = ["ADMIN"]
}

[nfx]

there could be a workaround to use external data resource, but https://github.com/databricks/cli/issues/695 really has to be fixed, as we don't always have an account ID.

provider "databricks" {
  alias      = "account"
  host       = module.defaults.aws_account_console
  account_id = module.defaults.aws_databricks_account_id
}

data "external" "account_token" {
  program = ["databricks", "auth", "token",
    "--host", module.defaults.aws_account_console,
    "--account-id", module.defaults.aws_databricks_account_id]
}

provider "databricks" {
  alias = "workspace"
  host  = databricks_mws_workspaces.this.workspace_url
  token = data.external.account_token.result.access_token
}

[marchar91]

I am experiencing the same issue and did the very same three attempts, did you solve like specified in the last comment? it's a bit pushed honestly, I would prefer a more stable solution

[nfx]

@marchar91 last comment is just the only possible workaround today

[docwhat]

I have the same issue, except I am using a service principal, which means I don't have a username & password to fall back on.

This does not use databricks cli at all.

provider "databricks" {
  alias = "accounts"

  host          = var.databricks_accounts_url
  client_id     = var.databricks_accounts_oauth_client
  client_secret = var.databricks_accounts_oauth_secret
  account_id    = var.databricks_accounts_account_id
}

I get the error:

Error: cannot read mws workspaces: cannot read token: inner token: oauth2: "invalid_client" "Client authentication failed"