[ISSUE] changed wheel file library causes error on terraform apply

[fjakobs]

I have a wheel library configured and after updating the wheel code I'm no longer able to apply terraform changes. I keep getting:

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for databricks_cluster.this to include new values learned so far during apply, provider
│ "registry.terraform.io/databrickslabs/databricks" produced an invalid new value for .library: planned set element
│ cty.ObjectVal(map[string]cty.Value{"cran":cty.ListValEmpty(cty.Object(map[string]cty.Type{"package":cty.String, "repo":cty.String})),
│ "egg":cty.NullVal(cty.String), "jar":cty.NullVal(cty.String), "maven":cty.ListValEmpty(cty.Object(map[string]cty.Type{"coordinates":cty.String,
│ "exclusions":cty.List(cty.String), "repo":cty.String})), "pypi":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"package":cty.StringVal("pyyaml"),
│ "repo":cty.NullVal(cty.String)})}), "whl":cty.NullVal(cty.String)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Configuration

resource "databricks_dbfs_file" "wheel" {
    source = "${path.module}/dist/russian_losses_dashboard-0.0.1-py3-none-any.whl"
    path = "${data.databricks_current_user.me.home}/Terraform/${var.resource_prefix}/russian_losses_dashboard-0.0.1-py3-none-any.whl"
}

resource "databricks_dbfs_file" "main" {
    source = "${path.module}/russian_losses_dashboard/jobs/equipment_losses.py"
    path = "${data.databricks_current_user.me.home}/Terraform/${var.resource_prefix}/equipment_losses.py"
}

resource "databricks_cluster" "this" {
  cluster_name  = "${var.resource_prefix}-russian_losses"
  num_workers   = 1
  spark_version = data.databricks_spark_version.latest.id
  node_type_id  = data.databricks_node_type.smallest.id
  autotermination_minutes = 20

  library {
    whl = databricks_dbfs_file.wheel.dbfs_path
  }

  library {
    pypi {
      package = "pyyaml"
    }
  }  
}

Expected Behavior

The library should be replaced

Actual Behavior

Fails with the error shown above

Steps to Reproduce

1. create a python wheel file
2. Create databricks_dbfs_file resource to upload the wheel file
3. Configure a cluster with an embedded library pointing to the wheel on dbfs
4. terraform apply
5. Make a code change and update the wheel file (python ./setup.py bdist_wheel
6. terraform apply
7. Booom
8. second call to terraform apply passes

Terraform and provider versions

$ terraform version
Terraform v1.1.9
on linux_arm64
+ provider registry.terraform.io/databrickslabs/databricks v0.5.5

Debug Output

https://gist.github.com/fjakobs/17cdc8442efbf5b7b47f1736e34deb71

[fjakobs]

When I move the library out into a library resource then it works just fine:

resource "databricks_library" "wheel" {
  cluster_id = databricks_cluster.this.cluster_id
  whl = databricks_dbfs_file.wheel.dbfs_path
}

[rob-mccann]

I think this is notable when creating a cluster via a databricks_job as (as far as I can tell?) we can't use the databricks_library resource there so applies fail.

[pietern]

I spent some time on this issue and found the following:

The library schema is a set type. When TF performs comparison of a resource to determine 1) if it has changed during planning, or 2) if it is consistent with planning during apply, it performs a set comparison on this type. When attaching a debugger to Terraform itself (not the provider) when reproducing this issue, I see that during (1) the set comparison function looks at values where optional strings are nil, while during (2) the set comparison function attempts to compare nil to an empty string, and fails. That's why the returned error says "does not correlate with any element in actual". Because of the nil vs empty string comparison, the object equivalence function returns false, and TF decides that the value to be applied is inconsistent with the value observed during planning.

[nizanth]

I am facing the same issue with provider version 1.40.0. Unfortunately we are not able to use databricks_library as a work around because are using a job cluster. Any other approach worked for library deployment?