search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.29k stars 1.72k forks source link

"billing_project" in provider configuration ignored if GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set #17882

Open mkielar opened 5 months ago

mkielar commented 5 months ago

Community Note

Terraform Version

Terraform v1.7.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v5.25.0

Affected Resource(s)

google_storage_bucket_iam_member (and probably others)

Terraform Configuration

# In "example-project-1", create a Service Account.
provider "google" {

  # Each project manages its own billing.
  user_project_override = true
  billing_project       = "[MASKED-PROJECT-A]" # Project A
  project               = "[MASKED-PROJECT-A]" # Project A
  region                = "europe-west3"

  # Note I'm impersonating different SAs here and below. In my scenario, each GCP Project 
  # has its own, dedicated deployer SA.
  impersonate_service_account = "infrastructure-deployer@[MASKED-PROJECT-A].iam.gserviceaccount.com"
}

resource "google_service_account" "example_sa" {
  account_id   = "example-sa"
}

# In "example-project-2", bind that Service Account as Viever for some existing bucket.
provider "google" {
  alias = "p2"

  # Each project manages its own billing.
  user_project_override = true
  billing_project       = "[MASKED-PROJECT-B]" # Project B
  project               = "[MASKED-PROJECT-B]" # Project B
  region                = "europe-west3"

  # Note I'm impersonating different SAs here and above. In my scenario, each GCP Project 
  # has its own, dedicated deployer SA.
  impersonate_service_account = "infrastructure-deployer@[MASKED-PROJECT-B].iam.gserviceaccount.com"
}

resource "google_storage_bucket_iam_member" "sa_viewer_permission" {
  provider = google.p2

  bucket = "[MASKED-PROJECT-B]-data-source-[MASKED]" # Some existing bucket
  member = "serviceAccount:${google_service_account.example_sa.email}"
  role   = "roles/storage.objectViewer"
}

Debug Output

2024-04-17T14:04:51.027+0200 [INFO] backend/local: apply calling Apply 2024-04-17T14:04:51.027+0200 [DEBUG] Building and walking apply graph for NormalMode plan 2024-04-17T14:04:51.027+0200 [DEBUG] Resource state not found for node "google_storage_bucket_iam_member.sa_viewer_permission", instance google_storage_bucket_iam_member.sa_viewer_permission 2024-04-17T14:04:51.027+0200 [DEBUG] ProviderTransformer: "google_service_account.example_sa (expand)" (terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"] 2024-04-17T14:04:51.028+0200 [DEBUG] ProviderTransformer: "google_storage_bucket_iam_member.sa_viewer_permission (expand)" (terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"].p2 2024-04-17T14:04:51.028+0200 [DEBUG] ProviderTransformer: "google_storage_bucket_iam_member.sa_viewer_permission" (terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/hashicorp/google"].p2 2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "google_service_account.example_sa (expand)" references: [] 2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "google_storage_bucket_iam_member.sa_viewer_permission (expand)" references: [google_service_account.example_sa (expand)] 2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "google_storage_bucket_iam_member.sa_viewer_permission" references: [google_service_account.example_sa (expand)] 2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: [] 2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"].p2" references: [] 2024-04-17T14:04:51.028+0200 [DEBUG] Starting graph walk: walkApply 2024-04-17T14:04:51.029+0200 [DEBUG] created provider logger: level=debug 2024-04-17T14:04:51.029+0200 [INFO] provider: configuring client automatic mTLS 2024-04-17T14:04:51.031+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5"] 2024-04-17T14:04:51.036+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11096 2024-04-17T14:04:51.036+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:51.072+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: configuring server automatic mTLS: timestamp="2024-04-17T14:04:51.072+0200" 2024-04-17T14:04:51.077+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: plugin address: address=/tmp/plugin402184316 network=unix timestamp="2024-04-17T14:04:51.077+0200" 2024-04-17T14:04:51.077+0200 [DEBUG] provider: using plugin: version=5 2024-04-17T14:04:51.082+0200 [DEBUG] created provider logger: level=debug 2024-04-17T14:04:51.082+0200 [INFO] provider: configuring client automatic mTLS 2024-04-17T14:04:51.085+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5"] 2024-04-17T14:04:51.089+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11109 2024-04-17T14:04:51.089+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:51.124+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: configuring server automatic mTLS: timestamp="2024-04-17T14:04:51.124+0200" 2024-04-17T14:04:51.129+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: plugin address: address=/tmp/plugin3613715731 network=unix timestamp="2024-04-17T14:04:51.129+0200" 2024-04-17T14:04:51.129+0200 [DEBUG] provider: using plugin: version=5 2024-04-17T14:04:51.136+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Authenticating using DefaultClient...: tf_mux_provider="proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1777 @module=google tf_req_id=f33b28ba-688d-fce3-e6a8-e8c7a8b272b7 timestamp="2024-04-17T14:04:51.136+0200" 2024-04-17T14:04:51.137+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: @module=google tf_mux_provider="proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=f33b28ba-688d-fce3-e6a8-e8c7a8b272b7 @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1778 tf_rpc=ConfigureProvider timestamp="2024-04-17T14:04:51.136+0200" 2024-04-17T14:04:51.137+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success] 2024-04-17T14:04:51.159+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Authenticating using DefaultClient...: @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1777 tf_rpc=ConfigureProvider @module=google tf_mux_provider="proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=fc8319ed-5ec3-acdb-8d07-4dc9cd7a0a4b timestamp="2024-04-17T14:04:51.159+0200" 2024-04-17T14:04:51.159+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: tf_mux_provider="proto5server.Server" @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1778 tf_req_id=fc8319ed-5ec3-acdb-8d07-4dc9cd7a0a4b tf_rpc=ConfigureProvider @module=google tf_provider_addr=registry.terraform.io/hashicorp/google timestamp="2024-04-17T14:04:51.159+0200" 2024-04-17T14:04:51.159+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success] 2024-04-17T14:04:51.431+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com: tf_mux_provider="proto5server.Server" tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1667 tf_req_id=fc8319ed-5ec3-acdb-8d07-4dc9cd7a0a4b @module=google tf_provider_addr=registry.terraform.io/hashicorp/google timestamp="2024-04-17T14:04:51.431+0200" 2024-04-17T14:04:51.433+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Authenticating using DefaultClient... 2024-04-17T14:04:51.433+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email] 2024-04-17T14:04:51.434+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success] 2024-04-17T14:04:51.502+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_A].iam.gserviceaccount.com: @module=google tf_provider_addr=registry.terraform.io/hashicorp/google @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1667 tf_mux_provider="*proto5server.Server" tf_req_id=f33b28ba-688d-fce3-e6a8-e8c7a8b272b7 tf_rpc=ConfigureProvider timestamp="2024-04-17T14:04:51.502+0200" 2024-04-17T14:04:51.504+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Authenticating using DefaultClient... 2024-04-17T14:04:51.504+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email] 2024-04-17T14:04:51.505+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success] 2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com 2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches 2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "Service Usage" 2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches 2024-04-17T14:04:51.584+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "IAM" 2024-04-17T14:04:51.654+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_A].iam.gserviceaccount.com 2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches 2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "Service Usage" 2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches 2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "IAM" 2024-04-17T14:04:51.657+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] [transport] [server-transport 0xc001000b60] Closing: Server.Stop called 2024-04-17T14:04:51.657+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] [transport] [server-transport 0xc001000b60] loopyWriter exiting with error: transport closed by client 2024-04-17T14:04:51.657+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2024-04-17T14:04:51.659+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11096 2024-04-17T14:04:51.659+0200 [DEBUG] provider: plugin exited google_storage_bucket_iam_member.sa_viewer_permission: Creating... 2024-04-17T14:04:51.661+0200 [INFO] Starting apply for google_storage_bucket_iam_member.sa_viewer_permission 2024-04-17T14:04:51.661+0200 [DEBUG] google_storage_bucket_iam_member.sa_viewer_permission: applying the planned Create change 2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] matching ID [MASKED_PROJECT_B]-data-source-[MASKED] to regex (?P[^/]+). 2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Locking "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]" 2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Locked "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]" 2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG]: Retrieving policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]" 2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success] 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Retry Transport: starting RoundTrip retry loop 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Retry Transport: request attempt 0 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Google API Request Details: 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ---[ REQUEST ]--------------------------------------- 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: GET /storage/v1/b/[MASKED_PROJECT_B]-data-source-[MASKED]/iam?alt=json&optionsRequestedPolicyVersion=3 HTTP/1.1 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: storage.googleapis.com 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: Terraform/1.7.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-User-Project: [MASKED_PROJECT_B] 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ----------------------------------------------------- 2024-04-17T14:04:52.344+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Google API Response Details: 2024-04-17T14:04:52.344+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ---[ RESPONSE ]-------------------------------------- 2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 403 Forbidden 2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Length: 626 2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: no-cache, no-store, max-age=0, must-revalidate 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 17 Apr 2024 12:04:52 GMT 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Expires: Mon, 01 Jan 1990 00:00:00 GMT 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Pragma: no-cache 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: UploadServer 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin 2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Guploader-Uploadid: ABPtcPq7mokd--I0jt3pGsme3vdcsCDuC2C1HDbwdsJJGBlxpKsdQUk-t-yTOb-lzI4l05T5Dgddn5HHuQ 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: { 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "error": { 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "code": 403, 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).", 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "errors": [ 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: { 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).", 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "domain": "global", 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "reason": "forbidden" 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: } 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ] 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: } 2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: } 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ----------------------------------------------------- 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 403 with body: HTTP/2.0 403 Forbidden 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Length: 626 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: no-cache, no-store, max-age=0, must-revalidate 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 17 Apr 2024 12:04:52 GMT 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Expires: Mon, 01 Jan 1990 00:00:00 GMT 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Pragma: no-cache 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: UploadServer 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Guploader-Uploadid: ABPtcPq7mokd--I0jt3pGsme3vdcsCDuC2C1HDbwdsJJGBlxpKsdQUk-t-yTOb-lzI4l05T5Dgddn5HHuQ 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: { 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "error": { 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "code": 403, 2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).", 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "errors": [ 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: { 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).", 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "domain": "global", 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "reason": "forbidden" 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: } 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ] 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: } 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: } 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Retry Transport: Returning after 1 attempts 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Unlocking "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]" 2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Unlocked "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]" 2024-04-17T14:04:52.349+0200 [ERROR] provider.terraform-provider-google_v5.25.0_x5: Response contains error diagnostic: @module=sdk.proto tf_req_id=b0c0c6be-5ba1-b287-368e-9557d3135e3d diagnostic_summary="Error retrieving IAM policy for storage bucket \"b/[MASKED_PROJECT_B]-data-source-[MASKED]\": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden" tf_proto_version=5.4 tf_rpc=ApplyResourceChange tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_storage_bucket_iam_member @caller=github.com/hashicorp/terraform-plugin-go@v0.22.1/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" diagnostic_severity=ERROR timestamp="2024-04-17T14:04:52.344+0200" 2024-04-17T14:04:52.356+0200 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot 2024-04-17T14:04:52.356+0200 [ERROR] vertex "google_storage_bucket_iam_member.sa_viewer_permission" error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden ╷ │ Error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden │ │ with google_storage_bucket_iam_member.sa_viewer_permission, │ on main.tf line 34, in resource "google_storage_bucket_iam_member" "sa_viewer_permission": │ 34: resource "google_storage_bucket_iam_member" "sa_viewer_permission" { │ ╵ 2024-04-17T14:04:52.360+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] [transport] [server-transport 0xc000e1da00] Closing: Server.Stop called 2024-04-17T14:04:52.360+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] [transport] [server-transport 0xc000e1da00] loopyWriter exiting with error: transport closed by client 2024-04-17T14:04:52.360+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2024-04-17T14:04:52.364+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11109 2024-04-17T14:04:52.364+0200 [DEBUG] provider: plugin exited

Expected Behavior

I would expect the terraform apply to be able to create both, the Service Account in Project A, and it's binding in IAM Policy of the Bucket in Project B.

Actual Behavior

Terraform creates the SA in Project A, but fails to provision the binding in Project 2. The error message is:

│ Error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden │ │ with google_storage_bucket_iam_member.sa_viewer_permission, │ on main.tf line 34, in resource "google_storage_bucket_iam_member" "sa_viewer_permission": │ 34: resource "google_storage_bucket_iam_member" "sa_viewer_permission" {

I believe the issue is a result of terraform provider implementation ignoring the billing_project setting on the provider whenever the GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set.

Steps to reproduce

First, a bit of explanation: I'm working on multiple GCP Projects (and credentials) at the same time, and it's easier for me to configure gcloud command line tool, as well as all tools using GCloud SDK (like terraform), using Environment Variables, rather than through gcloud auth login and gcloud auth application-default login. This way I can have multiple terminal sessions opened an operate with different credentials / gcp projects in each terminal.

Now to the steps:

  1. Following environment variables are set when running terraform apply 
    GOOGLE_CLOUD_PROJECT:                        [MASKED-PROJECT-A]
GOOGLE_APPLICATION_CREDENTIALS:              /home/mkielar/.config/gcloud/configurations/config_[MASKED-PROJECT-A].adc.json
GOOGLE_CLOUD_CPP_USER_PROJECT:               [MASKED-PROJECT-A]
GOOGLE_CLOUD_QUOTA_PROJECT:                  [MASKED-PROJECT-A]
  2. terraform apply (fails)

This causes the error as logged in the "Debug output":

│ Error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden │ │ with google_storage_bucket_iam_member.sa_viewer_permission, │ on main.tf line 34, in resource "google_storage_bucket_iam_member" "sa_viewer_permission": │ 34: resource "google_storage_bucket_iam_member" "sa_viewer_permission" {

However, when I unset the GOOGLE_CLOUD_QUOTA_PROJECT Environment Variable, everything works:

  1. Updated environment: 
    GOOGLE_CLOUD_PROJECT:                        [MASKED-PROJECT-A]
GOOGLE_APPLICATION_CREDENTIALS:              /home/mkielar/.config/gcloud/configurations/config_[MASKED-PROJECT-A].adc.json
GOOGLE_CLOUD_CPP_USER_PROJECT:               [MASKED-PROJECT-A]
  2. terraform apply (succeeds)

Important Factoids

  1. See "Actual behavior" and "Steps to reproduce".
  2. According to https://github.com/googleapis/google-cloud-php/blob/main/AUTHENTICATION.md#project-and-credential-lookup, the order in which GCloud PHP SDK (I know, Terraform uses Go, but they should be consistent), is that the priority is given to values defined in code, and only when they're missing in code, Environment is being considered. What terraform-provider-google is doing when GOOGLE_CLOUD_QUOTA_PROJECT suggests a reversed order - first, the Environment is considered, and if it's set, the value of billing_project in provider configuration is ignored.

References

No response

mkielar commented 5 months ago

Few more words about the "real-world" scenario:

In this case, I work predominantly on Project A, which is why I have it set up in Environment Variables. This setup allowed me to use gcloud cli, Python SDK and Terraform, while using dedicated Service Accounts for Terraform on each GCP Project (least-priviledge principle) while running it both locally and on our CICD Platform, without any changes in code.

The Terraform template in this case provisions around 60 other resources, and it does so predominantly in PROJECT A. The resources in PROJECT B (and C, and D, and so on) are mostly - as shown - IAM Policy Bindings, to allow Compute Resources in Project A, to access Storage Resources (GCS Buckets, BigQuery Tables) in Projects B, C, D...

Having things configured via Environment Variables allows me to use tools other than terraform (and these just operate on Project A, because that's like 99% of work), and get them billed correctly to PROJECT A billing.

ggtisc commented 5 months ago

Hi @mkielar!

Please confirm if both projects have the next same attributes with yes or no:

mkielar commented 4 months ago

Hi, @ggtisc , sorry for late reply. The thing is, I don't have access to entire GCP Organization. In fact, I can only see the two projects, and so far I used to set billing project to be the same as the project I'm operating on (so no shared billing project that I'd be aware of). all of that worked when I used gcloud CLI or when I used Terraform with just a single GCP Provider instance.

So, here are the answers, to my best knowledge:

SarahFrench commented 4 months ago

👋 Hi @mkielar , I noticed this issue by chance and my spidey senses are making me wonder if this is a result of some changes to the provider configuration code that we introduced in later v4.x.x versions. Your issue description mentions you're using a v5.x.x version, but could you please try reproducing the scenario using:

If the problem doesn't occur with v4.52.0 and does occur with v4.60.0 then that'll confirm the code changes I mentions are the origin of this problem. Also, your config from this issue will be a nice starting point for an acceptance test we can add to avoid a regression in future. Thanks!

mkielar commented 4 months ago

Hi @SarahFrench, I finally had some time over the weekend to test this, and here are the results:

All of the failing ones work when I unset the GOOGLE_CLOUD_QUOTA_PROJECT environment variable.

Hope that helps.

SarahFrench commented 4 months ago

Thank you for checking, given what you found I think it is related to the changes I mentioned.

SarahFrench commented 2 weeks ago

I've been looking into this a bit more and some stuff I found:

I'll have a go using the reproduction info in the description to reproduce the problem and test the different versions mentioned here, as there isn't a clear reason why the resources being provisioned in the reproduction above should be impacted by the muxing (as the reproduction uses no plugin-framework implemented resources).

It'll be useful to create an acceptance test defining the expected behaviour, but I think a fix for the changed behaviour might come as a side effect of https://github.com/GoogleCloudPlatform/magic-modules/pull/11577