search

databricks / terraform-provider-databricks

Databricks Terraform Provider
https://registry.terraform.io/providers/databricks/databricks/latest
Other
454 stars 391 forks source link

[ISSUE] Provisioning Databricks workspaces on GCP fails with a BAD Request #1321

Closed sstubben closed 2 years ago

sstubben commented 2 years ago

Hi there,

I'm trying to provision databricks on GCP, and has been given a preview document of how this should work by Databricks. The steps include setting up a databricks provisioning service account in one module, which works fine, and then adding this service account to Databricks: https://drive.google.com/file/d/1iAf0IKqR6C-QQFcmiFgOXJJm_KUcQrDJ/view?usp=sharing

Then I've created another module for actually setting up the databricks workspace. This module is given in Configuration below. For test purposes, I've defined some locals that will be moved to vars, once it works. The databricks_account_id, is as per my user and not the service account I've set up - as I am logged into the portal with my user: https://drive.google.com/file/d/1pUv0BiZYZpT_5qNkxujCJFqz6ggPjm7d/view?usp=sharing

The databricks_google_service_account is the service account created beforehand, with the following permissions:

iam.roles.create
iam.roles.delete
iam.roles.get
iam.roles.update
iam.serviceAccounts.getIamPolicy
iam.serviceAccounts.setIamPolicy
resourcemanager.projects.get
resourcemanager.projects.getIamPolicy
resourcemanager.projects.setIamPolicy
serviceusage.services.enable
serviceusage.services.get
serviceusage.services.list

The google_project is the project where databricks is enabled.

Configuration

terraform {
  required_providers {
    databricks = {
      source = "databrickslabs/databricks"
    }
  }
}

locals {
  databricks_account_id = "8****c"
  databricks_google_service_account = "test****gserviceaccount.com"
  google_project = "lunar-ingest-sist-dev"
}

#***********************#
# DATABRICKS ACCOUNTS #
#***********************#

provider "databricks" {
  alias                  = "accounts"
  host                   = "https://accounts.gcp.databricks.com"
  google_service_account = local.databricks_google_service_account
}

#TODO
provider "google" {
  project = local.google_project
  region = "europe-west1"
  zone = "europe-west1-c"
}

data "google_client_openid_userinfo" "me" {

}

data "google_client_config" "current" {

}

resource "databricks_mws_workspaces" "this" {
  provider       = databricks.accounts
  account_id     = local.databricks_account_id
  workspace_name = "databricks-echo-terraform"
  location       = data.google_client_config.current.region
  cloud_resource_bucket {
    gcp {
      project_id = data.google_client_config.current.project
    }
  }
}

Expected Behavior

I would have expected this code to provision / set up a new Databricks workspace.

Actual Behavior

The workspace is not created and I'm getting the following error: │ Error: cannot create mws workspaces: BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support

Steps to Reproduce

  1. terraform apply

Terraform and provider versions

Terraform v1.2.0
on darwin_arm64
+ provider registry.terraform.io/databrickslabs/databricks v0.5.7
+ provider registry.terraform.io/hashicorp/google v4.21.0

Debug Output

2022-05-20T11:51:58.111+0200 [INFO]  backend/local: apply calling Apply
2022-05-20T11:51:58.111+0200 [DEBUG] Building and walking apply graph for NormalMode plan
2022-05-20T11:51:58.112+0200 [DEBUG] Resource state not found for node "databricks_mws_workspaces.this", instance databricks_mws_workspaces.this
2022-05-20T11:51:58.112+0200 [DEBUG] ProviderTransformer: "databricks_mws_workspaces.this (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databrickslabs/databricks"].accounts
2022-05-20T11:51:58.112+0200 [DEBUG] ProviderTransformer: "databricks_mws_workspaces.this" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databrickslabs/databricks"].accounts
2022-05-20T11:51:58.112+0200 [DEBUG] ProviderTransformer: "data.google_client_openid_userinfo.me (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"]
2022-05-20T11:51:58.112+0200 [DEBUG] ProviderTransformer: "data.google_client_config.current (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"]
2022-05-20T11:51:58.113+0200 [DEBUG] pruning unused provider["registry.terraform.io/databrickslabs/databricks"]
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "databricks_mws_workspaces.this (expand)" references: []
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "local.databricks_google_service_account (expand)" references: []
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "databricks_mws_workspaces.this" references: [local.databricks_account_id (expand) data.google_client_config.current (expand) data.google_client_config.current (expand)]
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: [local.google_project (expand)]
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/databrickslabs/databricks\"].accounts" references: [local.databricks_google_service_account (expand)]
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "data.google_client_openid_userinfo.me (expand)" references: []
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "data.google_client_config.current (expand)" references: []
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "local.databricks_account_id (expand)" references: []
2022-05-20T11:51:58.114+0200 [DEBUG] ReferenceTransformer: "local.google_project (expand)" references: []
2022-05-20T11:51:58.114+0200 [DEBUG] pruneUnusedNodes: data.google_client_openid_userinfo.me (expand) is no longer needed, removing
2022-05-20T11:51:58.115+0200 [DEBUG] Starting graph walk: walkApply
2022-05-20T11:51:58.116+0200 [DEBUG] created provider logger: level=debug
2022-05-20T11:51:58.116+0200 [INFO]  provider: configuring client automatic mTLS
2022-05-20T11:51:58.132+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.5.7/darwin_arm64/terraform-provider-databricks_v0.5.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.5.7/darwin_arm64/terraform-provider-databricks_v0.5.7]
2022-05-20T11:51:58.136+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.5.7/darwin_arm64/terraform-provider-databricks_v0.5.7 pid=15904
2022-05-20T11:51:58.137+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.5.7/darwin_arm64/terraform-provider-databricks_v0.5.7
2022-05-20T11:51:58.153+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: Databricks Terraform Provider (experimental)
2022-05-20T11:51:58.153+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: 
2022-05-20T11:51:58.153+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: Version 0.5.7
2022-05-20T11:51:58.153+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: 
2022-05-20T11:51:58.153+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2022-05-20T11:51:58.153+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: 
2022-05-20T11:51:58.153+0200 [INFO]  provider.terraform-provider-databricks_v0.5.7: configuring server automatic mTLS: timestamp=2022-05-20T11:51:58.153+0200
2022-05-20T11:51:58.182+0200 [DEBUG] provider: using plugin: version=5
2022-05-20T11:51:58.182+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: plugin address: address=/var/folders/sb/wryg1cvs097199l6bgptf83w0000gn/T/plugin786715183 network=unix timestamp=2022-05-20T11:51:58.182+0200
2022-05-20T11:51:58.202+0200 [DEBUG] created provider logger: level=debug
2022-05-20T11:51:58.202+0200 [INFO]  provider: configuring client automatic mTLS
2022-05-20T11:51:58.209+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.21.0/darwin_arm64/terraform-provider-google_v4.21.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/google/4.21.0/darwin_arm64/terraform-provider-google_v4.21.0_x5]
2022-05-20T11:51:58.210+0200 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/databrickslabs/databricks\"].accounts" changed the config value, but that value is unused
2022-05-20T11:51:58.212+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.21.0/darwin_arm64/terraform-provider-google_v4.21.0_x5 pid=15905
2022-05-20T11:51:58.212+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.21.0/darwin_arm64/terraform-provider-google_v4.21.0_x5
2022-05-20T11:51:58.233+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: configuring server automatic mTLS: timestamp=2022-05-20T11:51:58.233+0200
2022-05-20T11:51:58.255+0200 [DEBUG] provider.terraform-provider-google_v4.21.0_x5: plugin address: address=/var/folders/sb/wryg1cvs097199l6bgptf83w0000gn/T/plugin809017900 network=unix timestamp=2022-05-20T11:51:58.255+0200
2022-05-20T11:51:58.255+0200 [DEBUG] provider: using plugin: version=5
2022-05-20T11:51:58.274+0200 [INFO]  provider.terraform-provider-databricks_v0.5.7: Explicit and implicit attributes: google_service_account, host: timestamp=2022-05-20T11:51:58.274+0200
2022-05-20T11:51:58.334+0200 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/hashicorp/google\"]" changed the config value, but that value is unused
2022-05-20T11:51:58.335+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [INFO] Authenticating using DefaultClient...: timestamp=2022-05-20T11:51:58.335+0200
2022-05-20T11:51:58.335+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: timestamp=2022-05-20T11:51:58.335+0200
2022-05-20T11:51:58.335+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [INFO] Authenticating using DefaultClient...: timestamp=2022-05-20T11:51:58.335+0200
2022-05-20T11:51:58.335+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: timestamp=2022-05-20T11:51:58.335+0200
2022-05-20T11:51:58.335+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [DEBUG] Waiting for state to become: [success]: timestamp=2022-05-20T11:51:58.335+0200
2022-05-20T11:51:58.850+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [INFO] Terraform is using this identity: sist@lunar.app: timestamp=2022-05-20T11:51:58.850+0200
2022-05-20T11:51:58.853+0200 [DEBUG] provider.terraform-provider-google_v4.21.0_x5: 2022/05/20 11:51:58 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /var/folders/sb/wryg1cvs097199l6bgptf83w0000gn/T/plugin809017900->: use of closed network connection 
2022-05-20T11:51:58.855+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-05-20T11:51:58.860+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.21.0/darwin_arm64/terraform-provider-google_v4.21.0_x5 pid=15905
2022-05-20T11:51:58.860+0200 [DEBUG] provider: plugin exited
2022-05-20T11:51:58.865+0200 [WARN]  Provider "registry.terraform.io/databrickslabs/databricks" produced an invalid plan for databricks_mws_workspaces.this, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .is_no_public_ip_enabled: planned value cty.True for a non-computed attribute
2022-05-20T11:51:58.866+0200 [INFO]  Starting apply for databricks_mws_workspaces.this
2022-05-20T11:51:58.867+0200 [DEBUG] databricks_mws_workspaces.this: applying the planned Create change
2022-05-20T11:51:58.870+0200 [INFO]  provider.terraform-provider-databricks_v0.5.7: Configured google-accounts auth: host=https://accounts.gcp.databricks.com, google_service_account=test****gserviceaccount.com: timestamp=2022-05-20T11:51:58.870+0200
2022-05-20T11:51:59.736+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: POST /api/2.0/accounts/8****c/workspaces {
  "account_id": "8****c",
  "cloud": "gcp",
  "cloud_resource_bucket": {
    "gcp": {
      "project_id": "lunar-ingest-sist-dev"
    }
  },
  "location": "europe-west1",
  "workspace_name": "databricks-echo-terraform"
}: timestamp=2022-05-20T11:51:59.736+0200
2022-05-20T11:52:00.366+0200 [DEBUG] provider.terraform-provider-databricks_v0.5.7: 400 Bad Request: timestamp=2022-05-20T11:52:00.365+0200
2022-05-20T11:52:00.366+0200 [WARN]  provider.terraform-provider-databricks_v0.5.7: /api/2.0/accounts/83c9f95f-9e80-432e-956b-c0b971daf11c/workspaces:400 - BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support: timestamp=2022-05-20T11:52:00.366+0200
2022-05-20T11:52:00.366+0200 [WARN]  provider.terraform-provider-databricks_v0.5.7: /api/2.0/accounts/83c9f95f-9e80-432e-956b-c0b971daf11c/workspaces:400 - BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support: timestamp=2022-05-20T11:52:00.366+0200
2022-05-20T11:52:00.391+0200 [ERROR] vertex "databricks_mws_workspaces.this" error: cannot create mws workspaces: BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support
╷
│ Error: cannot create mws workspaces: BAD_REQUEST: workspace creation REST API is not available to use. Please contact Databricks support
│ 
│   with databricks_mws_workspaces.this,
│   on main.tf line 41, in resource "databricks_mws_workspaces" "this":
│   41: resource "databricks_mws_workspaces" "this" {
│ 
╵
2022-05-20T11:52:00.414+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-05-20T11:52:00.416+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.5.7/darwin_arm64/terraform-provider-databricks_v0.5.7 pid=15904
2022-05-20T11:52:00.416+0200 [DEBUG] provider: plugin exited

Important Factoids

Are there anything atypical about your accounts that we should know? --> I don't think so 😄

nfx commented 2 years ago

@sstubben Underlying API is in a private preview. Please contact Databricks account team (SA, CSE, AE) or support to enable it for you. error clearly said that ;-)

It looks like you did everything correct from tf side.