google_bigquery_dataset_access unable to handle service account emails with mixed/upper cases

kamal-1 commented 1 week ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.6 on darwin_arm64

provider registry.terraform.io/hashicorp/google v6.3.0

Affected Resource(s)

google_bigquery_dataset_access

Terraform Configuration

resource "google_bigquery_dataset_access" "bigquery_dataset_access" {
  for_each      = var.access_requests
  project       = each.value.project_id
  dataset_id    = each.value.dataset_id
  role          = each.value.role
  user_by_email = each.value.user_by_email
}

Debug Output

https://gist.github.com/kamal-1/49cd54c47b1733296b918c08f71d0bb7

Expected Behavior

Terraform apply should succeed after the access is granted to the service account.

Actual Behavior

Terraform apply completes with an error. Note that the access is granted but it fails while validation for service accounts having upper case or mixed case email addresses.

│ Error: Provider produced inconsistent result after apply │ │ When applying changes to google_bigquery_dataset_access.bigquery_dataset_access["project-READER-ENTERPRISE-REPORTS-@project.iam.gserviceaccount.com"], provider │ "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker.

2024-09-19T21:44:24.224-0700 [DEBUG] provider.terraform-provider-google_v6.3.0_x5: 2024/09/19 21:44:24 [DEBUG] Skipping item with role= "OWNER", looking for "READER") 2024-09-19T21:44:24.224-0700 [DEBUG] provider.terraform-provider-google_v6.3.0_x5: 2024/09/19 21:44:24 [DEBUG] Skipping item with userByEmail= , looking for "enterprise-reports@project.iam.gserviceaccount.com") 2024-09-19T21:44:24.224-0700 [DEBUG] provider.terraform-provider-google_v6.3.0_x5: 2024/09/19 21:44:24 [DEBUG] Skipping item with userByEmail= "ENTERPRISE-REPORTS@project.iam.gserviceaccount.com", looking for "enterprise-reports@project.iam.gserviceaccount.com")

Steps to reproduce

Create a service account with upper case email address
Use google_bigquery_dataset_access to assign a BigQuery role to the upper case service account email.
Run terraform apply. Fails with error Error: Provider produced inconsistent result after apply

Important Factoids

Using application default credentials to perform terraform plan and terraform apply
It works fine if the service account email is all lower case. Issue happens only for upper/mixed case service accounts.

References

There have been similar issues in the past. We were hoping that 17994 being the latest one, would have fixed the issue but even the latest version 6.3.0 has the issue.

https://github.com/hashicorp/terraform-provider-google/issues/17994 https://github.com/hashicorp/terraform-provider-google/issues/11985

b/369584826

kamal-1 commented 4 days ago

I recently discovered that it is not possible to create service accounts with all caps/mixed case via Google Cloud Console. However, it is possible to do that via gcloud CLI.

gcloud iam service-accounts create TEST-SERVICE-ACCOUNT --description "Creating service account with all caps" --display-name TEST-SERVICE-ACCOUNT

Output: Created service account [TEST-SERVICE-ACCOUNT]

ggtisc commented 2 days ago

According to terraform registry isn't possible to create a google_service_account with uppercase email address.

Apparently it is possible to create it through CLI but I don't have permission to confirm this.

kamal-1 commented 2 days ago

According to terraform registry isn't possible to create a google_service_account with uppercase email address.

Apparently it is possible to create it through CLI but I don't have permission to confirm this.

My comment here confirms this understanding. CLI allows the service accounts to be created with upper case.

https://github.com/hashicorp/terraform-provider-google/issues/19547#issuecomment-2368876451

hashicorp / terraform-provider-google