[ISSUE] '401 Unauthorized' error when refreshing state of an already created workspace during a terraform plan

achaussende commented 2 years ago

Hi there,

I was upgrading the provider from 0.3.7 to 0.4.X on an existing Terraform codebase that only manages Databricks workspaces creation and then, I had an issue when refreshing the terraform state while doing a terraform plan. Please note that the terraform plan command works fine in Databricks Terraform provider 0.3.7 but not in any of the 0.4.X version (even 0.4.6).

Configuration

Basic Terraform code for workspace creation.

resource "databricks_mws_credentials" "workspace_credentials" {
  account_id       = local.databricks_account_id
  role_arn         = local.databricks_control_access_role.arn
  credentials_name = "${local.databricks_workspace_name}-credentials"
}

resource "databricks_mws_storage_configurations" "workspace_storage_configuration" {
  account_id                 = local.databricks_account_id
  bucket_name                = local.root_storage_bucket
  storage_configuration_name = "${local.root_storage_bucket}-storage"
}

data "aws_security_group" "default" {
  name   = "default"
  vpc_id = local.vpc_id
}

resource "databricks_mws_networks" "workspace_networks" {
  account_id         = local.databricks_account_id
  network_name       = "${local.databricks_workspace_name}-network"
  subnet_ids         = local.subnets_ids
  vpc_id             = local.vpc_id
  security_group_ids = [data.aws_security_group.default.id]
}

resource "databricks_mws_workspaces" "workspace" {
  account_id      = local.databricks_account_id
  aws_region      = var.region
  workspace_name  = local.databricks_workspace_name
  deployment_name = local.databricks_workspace_name

  credentials_id           = databricks_mws_credentials.workspace_credentials.credentials_id
  storage_configuration_id = databricks_mws_storage_configurations.workspace_storage_configuration.storage_configuration_id
  network_id               = databricks_mws_networks.workspace_networks.network_id
}

PS: Add the databricks provider configuration

provider "databricks" {
  host    = "https://accounts.cloud.databricks.com"
  profile = "accounts"
}

Expected Behavior

What should have happened?

The terraform plan command should have completed without freezing nor errors.

Actual Behavior

What actually happened?

When doing the refreshing state part for databricks_mws_workspaces.workspace, the command seems to freeze. Actually, after turning debug logging, it seems to got a 401 Unauthorized when calling an API on the wrong host (see logs below). I can only kill the command and thus, get an error.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

terraform plan
Wait for refreshing state of the workspace and wait a moment
Interrupt command

Terraform and provider versions

Please paste the output of terraform version. If version of databricks provider is not the latest (https://github.com/databrickslabs/terraform-provider-databricks/releases), please make sure to use the latest one.

Terraform v0.14.11
+ provider registry.terraform.io/databrickslabs/databricks v0.4.6
+ provider registry.terraform.io/hashicorp/aws v3.73.0

Debug Output

Simple output

databricks_mws_networks.workspace_networks: Refreshing state... 
databricks_mws_credentials.workspace_credentials: Refreshing state...
databricks_mws_storage_configurations.workspace_storage_configuration: Refreshing state... 
databricks_mws_workspaces.tenant_workspace: Refreshing state...
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...

Error: cannot read mws workspaces: workspace https://REDACTED.cloud.databricks.com is not yet reachable: Databricks API (/api/2.0/token/list) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/guides/aws-workspace

Output with debug logging and formatting (I redacted some information)

TF_LOG=DEBUG terraform plan 2>&1 | grep databricks | sed -E 's/^.* plugin[^:]+: (.*)$/\1/'

databricks_mws_storage_configurations.workspace_storage_configuration: Refreshing state... [id=***ACCOUNT_ID***/***STORAGE_CONF_ID***]
Using Basic authentication from ~/.databrickscfg: timestamp=2022-01-24T17:26:25.299+0100
Configured databricks-cli auth: host=https://accounts.cloud.databricks.com, username=***My email***, password=***REDACTED***, profile=accounts: timestamp=2022-01-24T17:26:25.299+0100
GET /api/2.0/accounts/***ACCOUNT_ID***/storage-configurations/***STORAGE_CONF_ID***: timestamp=2022-01-24T17:26:25.299+0100
databricks_mws_credentials.workspace_credentials: Refreshing state... [id=***ACCOUNT_ID***/***WKS_CREDENTIALS_ID***]
GET /api/2.0/accounts/***ACCOUNT_ID***/credentials/***WKS_CREDENTIALS_ID***: timestamp=2022-01-24T17:26:25.366+0100
2022/01/24 17:26:25 [INFO] ReferenceTransformer: reference not found: "local.databricks_account_id"
2022/01/24 17:26:25 [INFO] ReferenceTransformer: reference not found: "local.databricks_workspace_name"
2022/01/24 17:26:25 [DEBUG] ReferenceTransformer: "databricks_mws_networks.workspace_networks" references: []
databricks_mws_networks.workspace_networks: Refreshing state... [id=***ACCOUNT_ID***/***WKS_NETWORKS_ID***]
GET /api/2.0/accounts/***ACCOUNT_ID***/networks/***WKS_NETWORKS_ID***: timestamp=2022-01-24T17:26:25.696+0100
200 OK  {
    "bucket_name": "workspace-root"
  "storage_configuration_name": "workspace-root-storage"
200 OK  {
200 OK  {
2022/01/24 17:26:29 [INFO] ReferenceTransformer: reference not found: "local.databricks_account_id"
2022/01/24 17:26:29 [INFO] ReferenceTransformer: reference not found: "local.databricks_workspace_name"
2022/01/24 17:26:29 [INFO] ReferenceTransformer: reference not found: "local.databricks_workspace_name"
2022/01/24 17:26:29 [DEBUG] ReferenceTransformer: "databricks_mws_workspaces.workspace" references: []
databricks_mws_workspaces.workspace: Refreshing state... [id=***ACCOUNT_ID***/3740842647014325]
GET /api/2.0/accounts/***ACCOUNT_ID***/workspaces/3740842647014325: timestamp=2022-01-24T17:26:29.398+0100
200 OK  {
Waiting for state to become: [success]: timestamp=2022-01-24T17:26:29.654+0100
GET /api/2.0/accounts/***ACCOUNT_ID***/workspaces/3740842647014325: timestamp=2022-01-24T17:26:29.654+0100
200 OK  {
Workspace is now running: timestamp=2022-01-24T17:26:29.900+0100
Creating client for host https://my-workspace.cloud.databricks.com based on host=https://accounts.cloud.databricks.com, username=***My email***, password=***REDACTED***, profile=accounts: timestamp=2022-01-24T17:26:29.900+0100
Using directly configured basic authentication: timestamp=2022-01-24T17:26:29.900+0100
Configured basic auth: host=https://my-workspace.cloud.databricks.com, username=***My email***, password=***REDACTED***: timestamp=2022-01-24T17:26:29.900+0100
GET /api/2.0/token/list: timestamp=2022-01-24T17:26:29.900+0100
401 Unauthorized: timestamp=2022-01-24T17:26:30.284+0100
/api/2.0/token/list:401 - Databricks API (/api/2.0/token/list) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/guides/aws-workspace https://docs.databricks.com/dev-tools/api/latest/token.html#list: timestamp=2022-01-24T17:26:30.285+0100
/api/2.0/token/list:401 - Databricks API (/api/2.0/token/list) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/guides/aws-workspace https://docs.databricks.com/dev-tools/api/latest/token.html#list: timestamp=2022-01-24T17:26:30.285+0100
workspace https://my-workspace.cloud.databricks.com is not yet reachable: Databricks API (/api/2.0/token/list) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/guides/aws-workspace: timestamp=2022-01-24T17:26:30.285+0100
GET /api/2.0/accounts/***ACCOUNT_ID***/workspaces/3740842647014325: timestamp=2022-01-24T17:26:30.786+0100
200 OK  {
Workspace is now running: timestamp=2022-01-24T17:26:31.575+0100
Creating client for host https://my-workspace.cloud.databricks.com based on host=https://accounts.cloud.databricks.com, username=***My email***, password=***REDACTED***, profile=accounts: timestamp=2022-01-24T17:26:31.575+0100
Using directly configured basic authentication: timestamp=2022-01-24T17:26:31.575+0100
Configured basic auth: host=https://my-workspace.cloud.databricks.com, username=***My email***, password=***REDACTED***: timestamp=2022-01-24T17:26:31.575+0100
GET /api/2.0/token/list: timestamp=2022-01-24T17:26:31.575+0100
401 Unauthorized: timestamp=2022-01-24T17:26:32.092+0100
/api/2.0/token/list:401 - Databricks API (/api/2.0/token/list) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/guides/aws-workspace https://docs.databricks.com/dev-tools/api/latest/token.html#list: timestamp=2022-01-24T17:26:32.092+0100

***Then it cycles like that until I interrupt the command***

Important Factoids

Are there anything atypical about your accounts that we should know?

We use the same codebase to create multiple workspaces for our different environments. The command works for some of the workspaces (that are also already created and running).
Authentication to Accounts API (to create workspaces) is done using the ~/.databrickscfg file.
But I also tried using direct authentication in the Terraform code and it didn't work either
Those workspaces were created using Terraform provider 0.3.7 some time ago (like months)

nfx commented 2 years ago

@achaussende can you try auth with env vars instead of databricks-cli? in real production deployments you probably won't use ~/.databrickscfg file and rely on env vars.

achaussende commented 2 years ago

I think I find my problem and it is not related to how the provider works.

The workspace was not created using the same credentials (user/pw) I am currently using during my terraform plan. But during the plan, we are reusing the same credentials for both querying the Accounts/Admin console API (host=https://accounts.cloud.databricks.com) and the workspace's API (host=https://my-workspace.cloud.databricks.com).

Using env vars and changing to another user (like a service account to us, that is created in both admin console and that I suspected to be used for workspace creation and thus, that should be present in workspace's users) instead of my personal access allows the terraform plan command to complete.

How can I improve the authentication for our workspace lifecycle so we can manage it using Terraform only? Should we use the token output from resource.databricks_mws_workspaces?

nfx commented 2 years ago

@achaussende username/password could be used for both accounts & workspace API communication, there's no difference.

databricks_mws_workspaces.token is just a PAT token for the same user you're calling workspace creation APIs with.

achaussende commented 2 years ago

@nfx I know that both can be used for both API communication, but the credentials have to be created in both and i think, it's not that a good practice to reuse the same credentials to authenticate to the Admin console and to a workspace.

I am wondering if there is a better way so I don't have to reuse the same credentials for both API that the provider is querying.

nfx commented 2 years ago

@achaussende then you should use databricks_mws_workspaces.this.token[0].token_value. but it's perfectly fine to reuse the same credentials for both account and things within the workspace.

databricks / terraform-provider-databricks