GitHub CI fails shared env setup in 3-networks-dual-svpc without SA impersonation

[klondikedragon]

TL;DR

The GitHub CI readme, 3-networks-dual-svpc setup, Step 16+ (deploying the shared net env) expects the user logged in at the command line to be able to impersonate the sa-terraform-net@prj-b-seed-xxxx.iam.gserviceaccount.com service account. The 0-bootstrap project (GitHub variant) creates this service account, but it does not grant impersonation to the org admins group.

Expected behavior

Since service account impersonation is required to complete the GitHub CI instructions, 0-bootstrap GitHub module should have also added the proper impersonation rights.

Observed behavior

Step 17 (./tf-wrapper.sh init shared) failed because the logged in user failed to impersonate the service account.

Terraform Configuration

(N/A)

Terraform Version

(N/A)

Additional information

At first glance, it was confusing why impersonation rights weren't already created, because we are passing sa_enable_impersonation=true to the seed_bootstrap module in 0-bootstrap. Upon further investigation, it looks like previously the seed_bootstrap module was creating a single terraform service account and was adding impersonation rights. However, because create_terraform_sa is false (which makes sense, 0-bootstrap is creating granular service accounts instead), the seed_bootstrap module doesn't know about the granular service accounts and does not try to add impersonation rights to them.

This can be resolved by adding this code to the end of 0-bootstrap/github.tf:

# Certain steps still require manually running terraform from the command line (e.g., 3-networks-dual-svpc for shared env).
# Provide a "break glass" capability for organization administrators to impersonate the granular service accounts.

resource "google_service_account_iam_member" "org_admin_terraform_sa_user" {
  for_each = local.granular_sa

  service_account_id = google_service_account.terraform-env-sa[each.key].id
  role               = "roles/iam.serviceAccountUser"
  member             = "group:${var.groups.required_groups.group_org_admins}"
}

resource "google_service_account_iam_member" "org_admin_terraform_sa_impersonate_permissions" {
  for_each = local.granular_sa

  service_account_id = google_service_account.terraform-env-sa[each.key].id
  role               = "roles/iam.serviceAccountTokenCreator"
  member             = "group:${var.groups.required_groups.group_org_admins}"
}

If other CI flavors require terraform to be run manually at any point from the commandline, then similar changes would be needed for those flavors in the bootstrap step as well. Or even better, make this behavior parameterized and have the guides indicate the relevant flag needs to be turned on.

I'm sharing this in case it helps others trying to use this framework in conjunction with GitHub CI.