[ISSUE] Panic installing library on a `job_cluster` with `databricks_job` resource

[8sdhh290]

Configuration

resource "databricks_job" "example-name" {
  name = "example-name"
  job_cluster {
    job_cluster_key = "example-job-cluster"
    new_cluster {
      library {
        pypi {
          package = "example-pkg"
          repo    = "https://example.com/pypi/simple/"
        }
      }
      num_workers   = 1
      spark_version = data.databricks_spark_version.latest_lts.id
      node_type_id  = data.databricks_node_type.smallest.id
    }
  task {
    task_key        = "example-task"
    job_cluster_key = "example-job-cluster"
    notebook_task {
      notebook_path = "example.py"
    }
  }
  }

Expected Behavior

I expect the custom library to be installed on the job cluster.

Actual Behavior

On terraform apply I get a panic: Error: panic: interface conversion: interface {} is nil, not map[string]interface {} If I comment out the library block, terraform apply runs properly.

Steps to Reproduce

1. terraform apply

Terraform and provider versions

Terraform v1.9.8
on windows_amd64
+ provider registry.terraform.io/databricks/databricks v1.55.0

Is it a regression?

I have not used previous versions.

Debug Output

https://gist.github.com/8sdhh290/bcece748f09d0175719c26aa3f4f3fb1

The names are different that in the example as above, but the crash still holds.

Important Factoids

N/A

Would you like to implement a fix?

[alexott]

For jobs, libraries aren't installed on the cluster itself but attached to tasks... It's mentioned in the docs that not all parameters are supported: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job#new_cluster, although library not in the list.

[8sdhh290]

Thanks!