[ISSUE] Issue with data `databricks_cluster_policy`

[qnix-databricks]

Configuration

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}

provider "databricks" {
  config_file = "/Users/quan.ta/project/tf/terraform-databricks-examples/.databrickscfg"
  profile = "DEFAULT"
}

# The content of my config_file reference this workspace: 
# host=https://cse2.cloud.databricks.com

data "databricks_cluster_policy" "base" {
    # name = "Personal Compute"
    # name = "Shared Compute"
    # name = "Job Compute"
    # name = "Power User Compute"
    name = "Legacy Shared Compute"
}

output "cluster_policy_name" {
    value = data.databricks_cluster_policy.base.name
}

output "cluster_policy_family_id" {
    value = data.databricks_cluster_policy.base.policy_family_id
}

Expected Behavior

The code should return the cluster policy information for all valid default cluster policy family available in the workspace.

Actual Behavior

In my test, it appears only to work when name = "Personal Compute", and it return an error (as captured in the tf-debug.log) when other name value is used.

Steps to Reproduce

1. terraform plan

Terraform and provider versions

% terraform -version Terraform v1.9.7 on darwin_arm64

• provider registry.terraform.io/databricks/databricks v1.52.0

Is it a regression?

Same error in v1.50.0, v1.48.0, v1.40.0, v1.20.0

I stopped at v1.20.0. It seems this hasn't worked in quite many versions

Debug Output

tf-debug.log

Important Factoids

This bug is affecting PS deliverables for Publicis.

Would you like to implement a fix?

[alexott]

Works just fine for me:

locals {
  names = [
    "Personal Compute",
    "Shared Compute",
    "Job Compute",
    "Power User Compute",
    "Legacy Shared Compute"
  ]
}

data "databricks_cluster_policy" "base" {
  for_each = toset(local.names)
  name = each.value
}

output "cluster_policy_family_ids" {
  value = {
    for k, bd in data.databricks_cluster_policy.base : bd.name => bd.policy_family_id
  }
}

gives expected output:

Outputs:

cluster_policy_family_ids = {
  "Job Compute" = "job-cluster"
  "Legacy Shared Compute" = "shared-data-science"
  "Personal Compute" = "personal-vm"
  "Power User Compute" = "power-user"
  "Shared Compute" = "shared-compute"
}

[qnix-databricks]

Using the same code as yours, I got the following output

% tf plan
data.databricks_cluster_policy.base["Job Compute"]: Reading...
data.databricks_cluster_policy.base["Legacy Shared Compute"]: Reading...
data.databricks_cluster_policy.base["Shared Compute"]: Reading...
data.databricks_cluster_policy.base["Personal Compute"]: Reading...
data.databricks_cluster_policy.base["Power User Compute"]: Reading...
data.databricks_cluster_policy.base["Personal Compute"]: Read complete after 0s [id=E0631F5C0D00079B]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: cannot read cluster policy: cannot read data cluster policy: Policy named 'Job Compute' does not exist
│
│   with data.databricks_cluster_policy.base["Job Compute"],
│   on cluster_policies.tf line 27, in data "databricks_cluster_policy" "base":
│   27: data "databricks_cluster_policy" "base" {
│
╵
╷
│ Error: cannot read cluster policy: cannot read data cluster policy: Policy named 'Shared Compute' does not exist
│
│   with data.databricks_cluster_policy.base["Shared Compute"],
│   on cluster_policies.tf line 27, in data "databricks_cluster_policy" "base":
│   27: data "databricks_cluster_policy" "base" {
│
╵
╷
│ Error: cannot read cluster policy: cannot read data cluster policy: Policy named 'Power User Compute' does not exist
│
│   with data.databricks_cluster_policy.base["Power User Compute"],
│   on cluster_policies.tf line 27, in data "databricks_cluster_policy" "base":
│   27: data "databricks_cluster_policy" "base" {
│
╵
╷
│ Error: cannot read cluster policy: cannot read data cluster policy: Policy named 'Legacy Shared Compute' does not exist
│
│   with data.databricks_cluster_policy.base["Legacy Shared Compute"],
│   on cluster_policies.tf line 27, in data "databricks_cluster_policy" "base":
│   27: data "databricks_cluster_policy" "base" {
│
╵

There is no error on "Personal Compute"

[qnix-databricks]

It might has to do with permission. I logged in the workspace and noted that I do not have access to Policies

[alexott]

I would say that it's something with your workspace - I tested the production workspace on Azure. I've also tested with few other workspaces where I have admin access.

Lack of permissions will explain this.

[qnix-databricks]

Thank you for checking, @alexott