Terraform Permissions Denied Issue (403) while creating GCP notification Alerts and channels in Specific Project.

[yeshwanth14]

Hi,

I am encountering issues when attempting to create Google Cloud Platform (GCP) notification channels and alerts using Terraform for a specific project. Despite having the same permissions (including being an owner) across all projects, and having all required Monitoring APIs enabled, I receive a 403 Forbidden error when working with this particular project. I have verified that I have set up a service account with the appropriate roles, including Monitoring Editor, Monitoring AlertPolicy Editor, and Monitoring NotificationChannel Editor and tried changing the terraform providers, yet the issue persists. (Attached the Terraform Debug Logs)

Interestingly, I am able to create notification channels and alerts in other projects without any issues. This problem has been ongoing for over 35 days. FYI i'm able to create them via GCP Console for this project (glan-common-apigee-test), Terraform import also works

I would appreciate any assistance or guidance on resolving this issue.

google_monitoring_notification_channel.basic: Creating... ╷ │ Error: Error creating NotificationChannel: googleapi: Error 403: Permission denied (or the resource may not exist). │ │ with google_monitoring_notification_channel.basic, │ on main.tf line 1, in resource "google_monitoring_notification_channel" "basic": │ 1: resource "google_monitoring_notification_channel" "basic" {

terraform logs:-

-------------------

project:- glan-common-apigee-test (not working)

2024-07-26T19:26:30.282+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Waiting for state to become: [success] 2024-07-26T19:26:30.584+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [INFO] Terraform is using this identity: mallam@glan.com 2024-07-26T19:26:30.585+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] parent context canceled, cleaning up batcher batches 2024-07-26T19:26:30.585+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Stopping batcher "Service Usage" 2024-07-26T19:26:30.585+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] parent context canceled, cleaning up batcher batches 2024-07-26T19:26:30.585+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Stopping batcher "IAM" 2024-07-26T19:26:30.599+0530 [WARN] Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_monitoring_notification_channel.basic, but we are tolerating it because it is using the legacy plugin SDK. The following problems may be the cause of any confusing errors from downstream operations:

• .enabled: planned value cty.True for a non-computed attribute google_monitoring_notification_channel.basic: Creating... 2024-07-26T19:26:30.599+0530 [INFO] Starting apply for google_monitoring_notification_channel.basic 2024-07-26T19:26:30.600+0530 [DEBUG] google_monitoring_notification_channel.basic: applying the planned Create change 2024-07-26T19:26:30.601+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Locking "stackdriver/notifications/glan-common-apigee-test" 2024-07-26T19:26:30.601+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Locked "stackdriver/notifications/glan-common-apigee-test" 2024-07-26T19:26:30.601+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Creating new NotificationChannel: map[string]interface {}{"displayName":"Test Notification Channel", "enabled":true, "labels":map[string]string{"email_address":"cloud@glan.com"}, "type":"email"} 2024-07-26T19:26:30.601+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Waiting for state to become: [success] 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Retry Transport: starting RoundTrip retry loop 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Retry Transport: request attempt 0 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:30 [DEBUG] Google API Request Details: 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: ---[ REQUEST ]-------------- 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: POST /v3/projects/glan-common-apigee-test/notificationChannels?alt=json HTTP/1.1 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Host: monitoring.googleapis.com 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: User-Agent: Terraform/1.9.2 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.38.0 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Content-Length: 120 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Content-Type: application/json 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Accept-Encoding: gzip 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: { 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "displayName": "Test Notification Channel", 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "enabled": true, 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "labels": { 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "email_address": "cloud@glan.com" 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: }, 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "type": "email" 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: } 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5 2024-07-26T19:26:30.602+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: --------------------------------- 2024-07-26T19:26:33.375+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:26:33 [DEBUG] Google API Response Details: 2024-07-26T19:26:33.375+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: ---[ RESPONSE ]------------ 2024-07-26T19:26:33.375+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: HTTP/2.0 403 Forbidden 2024-07-26T19:26:33.375+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

######################################################################################

project:- glan-sandbox-ai (worked)

2024-07-26T19:23:58.381+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Waiting for state to become: [success] 2024-07-26T19:23:58.692+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [INFO] Terraform is using this identity: mallam@glan.com 2024-07-26T19:23:58.693+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] parent context canceled, cleaning up batcher batches 2024-07-26T19:23:58.693+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Stopping batcher "Service Usage" 2024-07-26T19:23:58.693+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] parent context canceled, cleaning up batcher batches 2024-07-26T19:23:58.693+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Stopping batcher "IAM" 2024-07-26T19:23:58.705+0530 [WARN] Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_monitoring_notification_channel.basic, but we are tolerating it because it is using the legacy plugin SDK. The following problems may be the cause of any confusing errors from downstream operations:

• .enabled: planned value cty.True for a non-computed attribute google_monitoring_notification_channel.basic: Creating... 2024-07-26T19:23:58.705+0530 [INFO] Starting apply for google_monitoring_notification_channel.basic 2024-07-26T19:23:58.705+0530 [DEBUG] google_monitoring_notification_channel.basic: applying the planned Create change 2024-07-26T19:23:58.706+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Locking "stackdriver/notifications/seg-sandbox-ai" 2024-07-26T19:23:58.706+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Locked "stackdriver/notifications/seg-sandbox-ai" 2024-07-26T19:23:58.706+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Creating new NotificationChannel: map[string]interface {}{"displayName":"Test Notification Channel", "enabled":true, "labels":map[string]string{"email_address":"cloud@glan.com"}, "type":"email"} 2024-07-26T19:23:58.706+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Waiting for state to become: [success] 2024-07-26T19:23:58.706+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Retry Transport: starting RoundTrip retry loop 2024-07-26T19:23:58.706+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Retry Transport: request attempt 0 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:23:58 [DEBUG] Google API Request Details: 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: ---[ REQUEST ]--------------- 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: POST /v3/projects/glan-sandbox-ai/notificationChannels?alt=json HTTP/1.1 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Host: monitoring.googleapis.com 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: User-Agent: Terraform/1.9.2 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.38.0 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Content-Length: 120 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Content-Type: application/json 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Accept-Encoding: gzip 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: { 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "displayName": "Test Notification Channel", 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "enabled": true, 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "labels": { 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "email_address": "cloud@glan.com" 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: }, 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: "type": "email" 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: } 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5 2024-07-26T19:23:58.707+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: --------------------------------- 2024-07-26T19:24:03.913+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: 2024/07/26 19:24:03 [DEBUG] Google API Response Details: 2024-07-26T19:24:03.913+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: ---[ RESPONSE ]----------------- 2024-07-26T19:24:03.913+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: HTTP/2.0 200 OK 2024-07-26T19:24:03.913+0530 [DEBUG] provider.terraform-provider-google_v5.38.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

[yeshwanth14]

Any help would be appreciated to resolving this issue!!