[ISSUE] Issue with `databricks_sql_table`: configuration drift without changes ignoring terraform `lifecycle_rules`

[mansenfranzen]

This issue is related to #3378. Every terraform apply causes a change in databricks_sql_table given no change in configuration. This is fairly annoying given the fact that the change requires databricks compute to spin up to execute the change which roughly takes 5 minutes to complete while blocking subsequent steps in CI/CD.

This behavior persists even after adding terraform's lifecycle rule [ignore_changes]() which should ignore such external configuration drifts.

Configuration

resource "databricks_sql_table" "bronze_layer" {
  name     = "session_files"

  catalog_name = "default"
  schema_name  = "bronze"

  table_type              = "EXTERNAL"
  data_source_format      = "PARQUET"
  storage_location        = "XXX"
  storage_credential_name = "XXX"

  # prevent configuration drift on properties
  lifecycle {
    ignore_changes = all
  }
}

Expected Behavior

No change upon apply given no configuration changes

Actual Behavior

  ~ update in-place
Terraform will perform the following actions:
  # databricks_sql_table.bronze_layer will be updated in-place
  ~ resource "databricks_sql_table" "bronze_layer" {
        id                      = "XXX"
        name                    = "session_files"
      + properties              = (known after apply)
        # (6 unchanged attributes hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Terraform and provider versions

terraform= 1.8.3 databricks = 1.41.0

[mansenfranzen]

Quick update: Using databricks provider v1.47.0 respects terraform's lifecycle rule. However, you need to ignore all. Setting ignore_changes to [properties] is not sufficient.

[nkvuong]

@mansenfranzen there might be some properties that we need to ignore in TF, are there any changes shown in the plan or is it just shown as (known after apply)?

Could you also share the debug log as well, since that would show the properties being sent back from the server

[apamildner]

We are seeing this as well. I have a hunch this is related to the table being of type EXTERNAL. @nkvuong I tried to look at the TRACE logs, but they are not giving any information (the response body is truncated). Querying the API directly i.e

curl --request GET -H "Authorization: Bearer foobar" https://my-databricks-url.net/api/2.1/unity-catalog/tables/my_catalog.my_table

shows me, when my_catalog.my_table is of type EXTERNAL, I'm not getting any "properties" back in the response body.

Could it be that the provider isn't handling this edge case of EXTERNAL tables somehow? I'm unable to find in the databricks docs anything that mentions EXTERNAL tables and properties either.

Not sure why properties are marked as "computed" here, maybe that can be a clue somehow?

Edit: I did some more research here. I think the problem is related to the value being computed because the provider tries to add fields that are automatically added by Databricks here. Maybe the use of a Diff suppress function could be the way to go instead of using a computed field?

[svrijssel]

We are experiencing the same issue, particularly with external tables.