google_bigquery_dataset - infinite diff if `access` is specified with non-legacy IAM roles

jcanseco commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.14.0
+ provider registry.terraform.io/hashicorp/google v5.0.0

Affected Resource(s)

google_bigquery_dataset

Terraform Configuration Files

provider "google" {
  project = "cnrm-jcanseco-2"
}

resource "google_bigquery_dataset" "dataset" {
  dataset_id = "bigquerydatasetsampletf"
  access {
    role = "roles/bigquery.dataOwner"
    user_by_email = google_service_account.bqowner.email
  }
}

resource "google_service_account" "bqowner" {
  account_id = "bqowner"
}

Debug Output

https://gist.github.com/jcanseco/f51d951db4a30378a0183cef0d21de08

Expected Behavior

Running terraform plan after terraform apply should return an empty diff.

Actual Behavior

Running terraform plan after terraform apply returned a non-empty diff:

$ terraform plan 
google_service_account.bqowner: Refreshing state... [id=projects/cnrm-jcanseco-2/serviceAccounts/bqowner@cnrm-jcanseco-2.iam.gserviceaccount.com]
google_bigquery_dataset.dataset: Refreshing state... [id=projects/cnrm-jcanseco-2/datasets/bigquerydatasetsampletf]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_bigquery_dataset.dataset will be updated in-place
  ~ resource "google_bigquery_dataset" "dataset" {
        id                              = "projects/cnrm-jcanseco-2/datasets/bigquerydatasetsampletf"
        # (11 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - user_by_email = "bqowner@cnrm-jcanseco-2.iam.gserviceaccount.com" -> null
        }
      + access {
          + role          = "roles/bigquery.dataOwner"
          + user_by_email = "bqowner@cnrm-jcanseco-2.iam.gserviceaccount.com"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Steps to Reproduce

terraform apply the above configuration
terraform plan the above configuration
Notice that the diff is non-empty.

It seems the issue is that Terraform's set hashing function for access doesn't account for the fact the BigQuery API only returns legacy IAM roles (e.g. OWNER) even if users set access[].role to modern IAM roles (e.g. roles/bigquery.dataOwner). See the API documentation for access[].role for more info.

So it seems like one idea for a fix is to implement a set hashing function that accounts for the conversion done by the BigQuery API.

References

b/179080823

b/359595039

venkykuberan commented 3 years ago

This is a known behavior, Predefined roles that have equivalent basic roles are swapped by the API to their basic counterparts. refer the doc here

You can change your config to use Basic role OWNER or you include life_cycle block like below to avoid the permadiff

resource "google_bigquery_dataset" "dataset" {
  dataset_id                  = "sink_test"
  project                     = var.project_id
  location                    = "US"
  default_table_expiration_ms = 3600000
  access {
    role = "roles/bigquery.dataOwner"
    user_by_email = google_service_account.bqowner.email
  }
   lifecycle {
      ignore_changes = [access]
  }
}

jcanseco commented 3 years ago

Hey @venkykuberan, I understand that the API converts predefined roles to their basic counterparts. I'm asking if it's possible to avoid a diff in the case where the predefined role on the config already matches the basic role in the live state, since there isn't actually a semantic difference in this case.

venkykuberan commented 3 years ago

@rileykarson what are you thoughts ?

rileykarson commented 3 years ago

It's currently expected behaviour that only legacy roles can be specified: https://github.com/GoogleCloudPlatform/magic-modules/pull/4191

I could see a case for allowing it, but at the moment that would be an enhancement and not a bug.

jcanseco commented 3 years ago

Gotcha I see, is there anything I need to do here to change this to an enhancement and not a bug then?

Also, I would not consider this high priority, though it has caused confusion for the user since we didn't have a warning like https://github.com/GoogleCloudPlatform/magic-modules/pull/4191 (and even if we did, I'm sure users will still run into the issue anyway since the underlying API does accept the new roles). We can add a similar warning on our side in the meantime though.

kemalizing commented 1 year ago

Hi, I know this is kind of an old issue but I have a similar problem. the only difference is I'm already using the legacy roles and it is still showing diff after applying.

Here is the resource:

terraform {
  required_version = "1.3.6"
  required_providers {
    google      = "4.48.0"
  }
}

resource "google_bigquery_dataset" "self_service" {
  project                     = var.self_service_project
  dataset_id                  = "${var.tenant}_${var.stage}_self_service"
  friendly_name               = "Views for ${var.tenant} agent tool"
  description                 = "Views for ${var.tenant} agent tool"
  location                    = var.dataset_location
  default_table_expiration_ms = null
  delete_contents_on_destroy  = false
  labels                      = null

  access {
    role           = "READER"
    group_by_email = "gcp-${var.self_service_project}-${var.tenant}${var.stage}Agent@email.com"
  }
  access {
    role          = "OWNER"
    special_group = "projectOwners"
  }
}

and here is the plan:

  ~ resource "google_bigquery_dataset" "self_service" {
        id                              = "projects/<project-id>/datasets/test_pre_self_service"
        # (13 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - special_group = "projectOwners" -> null
        }
      + access {
          + group_by_email = "gcp-testpreAgent@email.com"
          + role           = "READER"
        }
      - access {
          - group_by_email = "gcp-testpreAgent@email.com" -> null
          - role           = "READER" -> null
        }
      + access {
          + role          = "OWNER"
          + special_group = "projectOwners"
        }
    }

Any help will be highly appreciated. Thanks

ggtisc commented 3 months ago

This issue still persists, even without making any changes to the resources there is a permadiff when it is attempting to run a terraform plan or terraform apply

hashicorp / terraform-provider-google