Destruction of `google_bigquery_dataset_iam_member` does not remove the underlying IAM membership from the dataset policy

[najose]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.7.5 on linux_amd64

• provider registry.terraform.io/hashicorp/google v5.40.0
• provider registry.terraform.io/hashicorp/google-beta v5.40.0

Affected Resource(s)

google_bigquery_dataset_iam_member

Terraform Configuration

terraform {
  required_version = ">= 1.7.2"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 6.0"
    }
  }
}

resource "google_service_account" "test_sa" {
  project      = "my-test-project"
  account_id   = "test-sa-todel"
  display_name = "SA to test big query IAM bug."
}

resource "google_bigquery_dataset_iam_member" "test-bq-dataset-policy" {
  project    = "my-test-project"
  dataset_id = "dataset_id"
  role       = "roles/bigquery.dataEditor"
  member     = google_service_account.test_sa.member
}

Debug Output

No response

Expected Behavior

On destroying the resources (the google_service_account and google_bigquery_dataset_iam_member) are destroyed and the underlying IAM membership for the service account is removed from the BigQuery dataset IAM policy.

Actual Behavior

On destroying the resources (the google_service_account and google_bigquery_dataset_iam_member) are destroyed and the underlying IAM membership for the deleted service account is not removed from the BigQuery dataset IAM policy.

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

b/365168686

[melinath]

I am able to reproduce this with the following steps:

1. Apply this configuration:

resource "google_service_account" "test-account" {
  account_id   = "tf-test-bq-iam-kajwerhr"
  display_name = "Bigquery Dataset IAM Testing Account"
}

resource "google_bigquery_dataset_iam_member" "member" {
  dataset_id = google_bigquery_dataset.dataset.dataset_id
  role     = "roles/bigquery.dataViewer"
  member   = "serviceAccount:${google_service_account.test-account.email}"
}

resource "google_bigquery_dataset" "dataset" {
  dataset_id = "example_dataset"
}

2. In BigQuery Explorer, expand your project and click on example_dataset in the sidebar. Click on "Sharing > Permissions" and expand the BigQuery Data Viewer section. Confirm that the tf-test-bq-iam-kajwerhr is there.
3. Comment out the iam member resource & reapply.
4. Reload the BigQuery permissions and confirm that the tf-test-bq-iam-kajwerhr account is still there.

I waited five minutes to be sure this wasn't somehow a propagation issue.

[melinath]

It looks like this may have been an intentional change as part of https://github.com/GoogleCloudPlatform/magic-modules/pull/11352 based on the release note:

bigquery: made `google_bigquery_dataset_iam_member` non-authoritative. To remove a bigquery dataset iam member, use an authoritative resource like `google_bigquery_dataset_iam_policy`

However, I agree that this is potentially a dangerous behavior (since it could silently leave permissions in place that a user thought were removed.) Forwarding to engineering for consideration.

[melinath]

Additional context: IAM member resources are generally considered "non-authoritative" in that they don't remove other parts of the IAM policy. However, they still remove the access described in the resource when deleted.

If this new behavior is intentional, it could be worth reconsidering the documentation to make it more clear that this is a different "non-authoritative" than other IAM member resources.

[bardsleysdgr]

@melinath thank you for examing this issue. I appreciate and understand your comments, however...

resource "google_bigquery_dataset_iam_member" "member" {
  dataset_id = google_bigquery_dataset.dataset.dataset_id
  role     = "roles/bigquery.dataViewer"
  member   = "serviceAccount:${google_service_account.test-account.email}"
}

Declaring this resource creates the IAM binding, it doesn't do anything else, that is all it does. Why would removing this resource not remove the binding?

Forcing people to "To remove a bigquery dataset iam member, use an authoritative resource like google_bigquery_dataset_iam_policy" creates an entirely new paradigm of the relationship between a resource declaration in HCL and the actual object. Where objects are created in one resource block but removed by creating an entirely new and different resource block that exists to define the absence of something that has been deleted.

This is a significant change in how both terraform and the google provider work, and effectively deprecates "IAM member" as a concept and forces everyone to use "iam binding" or "iam policy".

[oprudkyi]

we used next approach

• google_bigquery_dataset_iam_member used to give permissions (we can't use authoritative, because there are hundreds of root modules)
• removing permissions via deleting google_bigquery_dataset_iam_member

now we have huge security issue because deleting google_bigquery_dataset_iam_member doesn't remove real iam

[fpopic]

Is this issue resolved? can it be closed?

[melinath]

It looks like this was resolved by https://github.com/GoogleCloudPlatform/magic-modules/pull/11853