search

databricks / terraform-provider-databricks

Databricks Terraform Provider
https://registry.terraform.io/providers/databricks/databricks/latest
Other
445 stars 384 forks source link

[ISSUE] unable to retrieve `databricks_spark_version` data source #2652

Closed jtravell closed 1 year ago

jtravell commented 1 year ago

We are encountering two problems when we try to include a data source of type databricks_spark_version into our terraform module as follows:

Configuration

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}

provider "databricks" {
  account_id = var.databricks_account_id
  alias      = "mws"
  host       = "https://accounts.cloud.databricks.com"
  username   = var.databricks_account_username
  password   = var.databricks_account_password
}

data "databricks_spark_version" "latest" {}

#data "databricks_spark_version" "latest" {
#  long_term_support = true
#  depends_on = [
#     data.databricks_mws_workspaces.all
#  ]
#}
#data "databricks_mws_workspaces" "all" {}

#output "all_mws_workspaces" {
#  value = data.databricks_mws_workspaces.all.ids
#}

Expected Behavior

The expected behavior is for the code to run without error, so that other resources can use the contents of the populated data source without error as well.

Actual Behavior

Instead, we get either of two errors.

First, if we run the code exactly as above, we get an error as follows when we run terraform apply: 

data.databricks_spark_version.latest: Reading...
╷
│ Error: default auth: cannot configure default credentials
│
│   with data.databricks_spark_version.latest,
│   on providers.tf line 17, in data "databricks_spark_version" "latest":
│   17: data "databricks_spark_version" "latest" {}
│
╵

Second, when we try to resolve the error by commenting out line 17 in the above code and by uncommenting every line after line 17 in the above code to include a depends_on per this link in the documentation, the following alternative error is returned instead, even though you can clearly see in the code above that the account_id parameter is clearly included in the provider block: 

data.databricks_mws_workspaces.all: Reading...
╷
│ Error: cannot read data mws workspaces: provider block is missing `account_id` property
│
│   with data.databricks_mws_workspaces.all,
│   on providers.tf line 25, in data "databricks_mws_workspaces" "all":
│   25: data "databricks_mws_workspaces" "all" {}
│
╵

We confirmed that all required fields are present in the variables passed into the terraform module, and that all of the values are correct. So the value for var.databricks_account_id is indeed being passed in.

Steps to Reproduce

To reproduce these two errors, follow these steps:

  1. Get correct values for databricks_account_username, databricks_account_password, and databricks_account_id. We got the correct values by creating a new Databricks account and then validating the values by using the values to successfully instantiate this other module before we took the steps in this issue.
  2. Then switch to a different directory which contains the single code file with the code above and run terraform init
  3. Then run terraform apply in the same new directory to invoke the breaking code shown at the top of this issue. For this step, use the code with the comments as shown above. This will reproduce the first error.
  4. Then comment out line 17 above and also remove the # from all the lines after 17 to uncomment all lines after line 17.
  5. Then run terraform apply again to reproduce the second error.

Terraform and provider versions

terraform version
Terraform v1.5.6
on windows_amd64
+ provider registry.terraform.io/databricks/databricks v1.24.1

Debug Output

For the first error, the debug log is: 

PS C:\Users\user8> ${env:TF_LOG}
DEBUG
PS C:\p\a\datalakes\dbricks-sparks_version> C:\terraform\terraform.exe apply -var-file="workspace.tfvars"
2023-09-04T15:50:59.957-0700 [INFO]  Terraform version: 1.5.6

2023-09-04T15:51:00.530-0700 [DEBUG] provider.terraform-provider-databricks_v1.24.1.exe:
2023-09-04T15:51:00.530-0700 [DEBUG] provider.terraform-provider-databricks_v1.24.1.exe: https://registry.terraform.io/providers/databricks/databricks/latest/docs
2023-09-04T15:51:00.530-0700 [DEBUG] provider.terraform-provider-databricks_v1.24.1.exe:
2023-09-04T15:51:00.533-0700 [INFO]  provider.terraform-provider-databricks_v1.24.1.exe: configuring server automatic mTLS: timestamp=2023-09-04T15:51:00.531-0700
2023-09-04T15:51:00.575-0700 [DEBUG] provider: using plugin: version=5
2023-09-04T15:51:00.575-0700 [DEBUG] provider.terraform-provider-databricks_v1.24.1.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2023-09-04T15:51:00.569-0700
2023-09-04T15:51:00.617-0700 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/databricks/databricks\"]" changed the config value, but that value is unused
2023-09-04T15:51:00.621-0700 [INFO]  provider.terraform-provider-databricks_v1.24.1.exe: Explicit and implicit attributes:: timestamp=2023-09-04T15:51:00.620-0700
2023-09-04T15:51:00.622-0700 [DEBUG] Resource instance state not found for node "data.databricks_spark_version.latest", instance data.databricks_spark_version.latest
2023-09-04T15:51:00.622-0700 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest" references: []
data.databricks_spark_version.latest: Reading...
2023-09-04T15:51:00.631-0700 [ERROR] provider.terraform-provider-databricks_v1.24.1.exe: Response contains error diagnostic: diagnostic_detail= diagnostic_summary="default auth: cannot configure default credentials" tf_provider_addr=registry.terraform.io/databricks/databricks diagnostic_severity=ERROR tf_data_source_type=databricks_spark_version tf_proto_version=5.3 tf_req_id=22e5e1d3-11de-f7d2-4499-86d4b040b969 tf_rpc=ReadDataSource @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto timestamp=2023-09-04T15:51:00.631-0700
2023-09-04T15:51:00.632-0700 [ERROR] vertex "data.databricks_spark_version.latest" error: default auth: cannot configure default credentials
2023-09-04T15:51:00.632-0700 [ERROR] vertex "data.databricks_spark_version.latest (expand)" error: default auth: cannot configure default credentials
╷
│ Error: default auth: cannot configure default credentials
│
│   with data.databricks_spark_version.latest,
│   on providers.tf line 17, in data "databricks_spark_version" "latest":
│   17: data "databricks_spark_version" "latest" {}
│
╵
2023-09-04T15:51:00.638-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-09-04T15:51:00.670-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.24.1/windows_amd64/terraform-provider-databricks_v1.24.1.exe pid=62804
2023-09-04T15:51:00.671-0700 [DEBUG] provider: plugin exited

Then, for the second error, the debug log is: 

2023-09-04T16:07:54.418-0700 [INFO]  Terraform version: 1.5.6
2023-09-04T16:07:54.930-0700 [DEBUG] provider: using plugin: version=5
2023-09-04T16:07:54.930-0700 [DEBUG] provider.terraform-provider-databricks_v1.24.1.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2023-09-04T16:07:54.930-0700
2023-09-04T16:07:54.956-0700 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/databricks/databricks\"]" changed the config value, but that value is unused
2023-09-04T16:07:54.957-0700 [INFO]  provider.terraform-provider-databricks_v1.24.1.exe: Explicit and implicit attributes:: timestamp=2023-09-04T16:07:54.957-0700
2023-09-04T16:07:54.960-0700 [DEBUG] Resource instance state not found for node "data.databricks_mws_workspaces.all", instance data.databricks_mws_workspaces.all
2023-09-04T16:07:54.960-0700 [DEBUG] ReferenceTransformer: "data.databricks_mws_workspaces.all" references: []
data.databricks_mws_workspaces.all: Reading...
2023-09-04T16:07:54.964-0700 [ERROR] provider.terraform-provider-databricks_v1.24.1.exe: Response contains error diagnostic: tf_data_source_type=databricks_mws_workspaces tf_rpc=ReadDataSource diagnostic_detail= diagnostic_summary="cannot read data mws workspaces: provider block is missing `account_id` property" tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=c84c0dfe-2ef6-0180-f5b7-986c9a57cc4e @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_severity=ERROR timestamp=2023-09-04T16:07:54.964-0700
2023-09-04T16:07:54.965-0700 [ERROR] vertex "data.databricks_mws_workspaces.all" error: cannot read data mws workspaces: provider block is missing `account_id` property
2023-09-04T16:07:54.965-0700 [ERROR] vertex "data.databricks_mws_workspaces.all (expand)" error: cannot read data mws workspaces: provider block is missing `account_id` property
╷
│ Error: cannot read data mws workspaces: provider block is missing `account_id` property
│
│   with data.databricks_mws_workspaces.all,
│   on providers.tf line 25, in data "databricks_mws_workspaces" "all":
│   25: data "databricks_mws_workspaces" "all" {}
│
╵
2023-09-04T16:07:54.967-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-09-04T16:07:54.995-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databricks/databricks/1.24.1/windows_amd64/terraform-provider-databricks_v1.24.1.exe pid=62980
2023-09-04T16:07:54.996-0700 [DEBUG] provider: plugin exited

Important Factoids

We are NOT using environment variables because our use case prohibits use of environment variables. So we pass the databricks_account_username, databricks_account_password, and databricks_account_id variables in as terraform input variables per the code above.

nkvuong commented 1 year ago

databricks_spark_version require a workspace-level provider, and won't work with the account-level provider as your code is showing

jtravell commented 1 year ago

@nkvuong Can you please show the actual code required to implement the change that you suggest? And can you please show a link to documentation that explains the code that you will please show?