Logging Sink missing Storage Legacy Bucket Owner - specific to GCS bucket targets for the log sink

[obriensystems]

Issue creating GCS bucket sink targets - IAM permissions on the SA see #318

review central logging project bucket for centralized sink https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/audits/logging-project/cloud-logging-buckets.yaml#L41

On one of my organizations the service account get auto assigned On another clean org with only the TF LZ installed the SA is not assigned Creating a manual GCS log storage target for the sink does not set the SA permission

• non-default service accounts for GCS bucket targets of log sinks - https://cloud.google.com/logging/docs/routing/user-managed-service-accounts

• {
  errorGroups: [1]
  insertId: "1uvlr6ebn7"
  labels: {7}
  logName: "projects/tzpe-tlz-tlzprod-host4/logs/logging.googleapis.com%2Fsink_error"
  receiveTimestamp: "2023-11-02T16:10:31.566259282Z"
  resource: {2}
  severity: "ERROR"
  textPayload: "Cloud Logging sink configuration error in tzpe-tlz-tlzprod-host4, sink 20231015-prod-gcs-sink: bucket_permission_denied ()"
  timestamp: "2023-11-02T16:10:30.582419156Z"

  

Missing SA permissions on the Cloud Storage Bucket

see https://cloud.google.com/logging/docs/export/configure_export_v2#gcloud_3

missing

service-951469276805@gcp-sa-logging.iam.gserviceaccount.com	Cloud Logging Service Account for Project 951469276805	Storage Legacy Bucket Owner

get the cloud logging service account SA by determining the project number for the prod4 project

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L221

root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ PROJECT_ID=tzpe-tlz-tlzprod-host4
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ KCC_PROJECT_NUMBER=$(gcloud projects list --filter="${PROJECT_ID}" '--format=value(PROJECT_NUMBER)')
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ echo $KCC_PROJECT_NUMBER
604049845861

How does the cloud logging service account get auto assigned to the GCS bucket? https://cloud.google.com/logging/docs/buckets#which_service_accounts_are_routing_logs_to_my_bucket "Logs Bucket Writer" in roles off IAM permissions - none there on either org - even though "ol" has the SA

I think we need "Storage Legacy Bucket Owner"

SA is https://cloud.google.com/logging/docs/routing/user-managed-service-accounts

root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ gcloud logging settings describe --project=$PROJECT_ID
kmsServiceAccountId: cmek-p604049845861@gcp-sa-logging.iam.gserviceaccount.com
loggingServiceAccountId: service-604049845861@gcp-sa-logging.iam.gserviceaccount.com
name: projects/tzpe-tlz-tlzprod-host4/settings

and
root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ gcloud logging settings describe --project=$PROJECT_ID
kmsServiceAccountId: cmek-p604049845861@gcp-sa-logging.iam.gserviceaccount.com
loggingServiceAccountId: service-604049845861@gcp-sa-logging.iam.gserviceaccount.com
name: projects/tzpe-tlz-tlzprod-host4/settings

[obriensystems]

context terraform.landing.systems buckets via terraform are up

storage bucket and logging bucket created for 2 routers in bigquery-ol at the org scope 20231106:1630

1705

GCS entries up later

sink details

comparing

permissions

busted org

working org

[obriensystems]

Check existing config flag set to true before we swap in the service account - see if it kicks in the default sa around https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink

  # Use a unique writer (creates a unique service account used for writing)
  unique_writer_identity = true

change

root_@cloudshell:~/lz-tls/_lz2/pbmm-on-gcp-onboarding (lz-tls)$ git diff
diff --git a/modules/23-logging/main.tf b/modules/23-logging/main.tf
index 943f4b0..8472e25 100644
--- a/modules/23-logging/main.tf
+++ b/modules/23-logging/main.tf
@@ -69,7 +69,7 @@ resource "google_logging_project_sink" "prod-log-sink-to-gcs-bucket" {
   #filter = "severity >= INFO"

   # Use a unique writer (creates a unique service account used for writing)
-  unique_writer_identity = true
+  #unique_writer_identity = true

results - new SA serviceAccount:service-604049845861@gcp-sa-logging.iam.gserviceaccount.com

Step #3 - "tf plan": Terraform will perform the following actions:
Step #3 - "tf plan": 
Step #3 - "tf plan":   # module.project-level-log-sink.google_logging_project_sink.prod-log-sink-to-gcs-bucket must be replaced
Step #3 - "tf plan": -/+ resource "google_logging_project_sink" "prod-log-sink-to-gcs-bucket" {
Step #3 - "tf plan":       - disabled               = false -> null
Step #3 - "tf plan":       ~ id                     = "projects/tzpe-tlz-tlzprod-host4/sinks/20231015-prod-gcs-sink" -> (known after apply)
Step #3 - "tf plan":         name                   = "20231015-prod-gcs-sink"
Step #3 - "tf plan":       ~ unique_writer_identity = true -> false # forces replacement
Step #3 - "tf plan":       ~ writer_identity        = "serviceAccount:service-604049845861@gcp-sa-logging.iam.gserviceaccount.com" -> (known after apply)
Step #3 - "tf plan":         # (2 unchanged attributes hidden)
Step #3 - "tf plan": 
Step #3 - "tf plan":       + bigquery_options {
Step #3 - "tf plan":           + use_partitioned_tables = (known after apply)
Step #3 - "tf plan":         }
Step #3 - "tf plan":     }
Step #3 - "tf plan": 
Step #3 - "tf plan":   # module.service_accounts.data.template_file.keys["sa"] will be read during apply
Step #3 - "tf plan":   # (config refers to values not yet known)
Step #3 - "tf plan":  <= data "template_file" "keys"  {
Step #3 - "tf plan":       ~ id       = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" -> (known after apply)
Step #3 - "tf plan":       + rendered = (known after apply)
Step #3 - "tf plan":         # (2 unchanged attributes hidden)
Step #3 - "tf plan":     }
Step #3 - "tf plan": 

[obriensystems]

[fmichaelobrien]

20240406: Closing issue during retrofit/rebase of this TEF V1 based/modified repo to TEF V4 standards This issue may participate in the LZ refactor after rebase Query on all issues related to the older V1 version via the tag https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/labels/2024-pre-tef-v4