[ISSUE] Issue with `databricks_grants` resource - Permission granted does not match code, plan, or state file

[ACSmith07]

When running the databricks_grant resource, the incorrect group receives permissions. This happens only on 2 instances out of ~30, but happens consistently with those 2. The permissions are swapped between them. It is also nondeterministic. Some runs permissions end up correct, some they end up wrong.

The very concerning part is the incorrect settings are not represented in either terraform plan or the state file. We just find out when users report they've lost access.

Configuration

Code:

hcl
resource "databricks_grants" "gaming_analytics_scratchpad" {
  schema = databricks_schema.gaming_analytics_scratchpad.id
  grant {
      principal = "Databricks Secured - Gaming"
      privileges = var.schema_grant_presets["all"]
    }
}

Additional context in attached screenshots.

Expected Behavior

Group "Databricks Secured - Gaming" are granted ALL PERMISSIONS on the schema

Actual Behavior

A different group "Databricks Secured - HR" are granted ALL PERMISSIONS on the schema, and "Databricks Secured - Gaming" have no grants. However the state file does not reflect this.

The inverse also occurs - the HR schema periodically has grants for the Gaming group but none for the HR group. On subsequent runs this periodically corrects, then periodically drifts again.

State file:

{
      "module": "module.build_scratchpad[0]",
      "mode": "managed",
      "type": "databricks_grants",
      "name": "gaming_analytics_scratchpad",
      "provider": "provider[\"registry.terraform.io/databricks/databricks\"].workspace",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "catalog": null,
            "external_location": null,
            "foreign_connection": null,
            "function": null,
            "grant": [
              {
                "principal": "Databricks Secured - Gaming",
                "privileges": [
                  "ALL_PRIVILEGES"
                ]
              }
            ],

Screenshot with incorrect Grants:

Steps to Reproduce

Repeated runs of terraform apply with no changes to the relevant code.

Terraform and provider versions

Is it a regression?

No issues with ~30 other instances of the databricks_grants resource, and no issues with the Gaming instance until the HR instance was created.

Debug Output

Important Factoids

Would you like to implement a fix?

databricks_grants bug screenshots.docx

[ACSmith07]

Just ran this project again. terraform plan recognizes that the permissions are wrong for both impacted schemas:

The apply succeeds:

Result: permissions are wrong. Having no assigned group is a new behavior actually, it had previously reset correct, or stuck with HR (non-deterministic, wouldn't match terraform apply output).

[hectorcast-db]

Hello @ACSmith07

Thank you for reporting this. I am not able to reproduce this. It is very unusual for this to happen, since if there are no changes, and the resource is not part of the plan, TF won't update the resource. Could it be some other project/code that is changing the real(UI) status? Can you share the debug logs and exact steps to reproduce it?

Thank you

[ACSmith07]

@hectorcast-db Sorry for the slow response. I found I had a case sensitivity typo in the Gaming group here. The problem seems to have gone away. A bit hard to say since it was intermittent. Everywhere else I had the same exact typo (groups were created with Databricks/DataBricks interchangeably :/ ) the resource would fail. So this maybe points to a deeper issue where group names aren't being resolved correctly under some edge case? And that was causing the typo'd name to resolve inconsistently to Gaming vs HR (neither of which was exact match).

Feel free to close this one if it's too far afield at this point, and sorry for the noise.