Stage 1 failing with "One or more users named in the policy do not belong to a permitted customer"

[klondikedragon]

TL;DR

I'm following the GitHub CI readme, and working on deploying step 1. Applying step 1 first failed because of project quota errors. I have already requested and received an increase project quota, and cleaned up the terraform state. It's now progressing and creating the projects properly.

However, it's now failing to apply an IAM policy to the itl-c-logging project. It fails with "Error 400: One or more users named in the policy do not belong to a permitted customer."

Expected behavior

terraform apply should be successful.

Observed behavior

Terraform apply is failing with (billing account ID redacted):

Error: Request `Create IAM Members roles/logging.logWriter serviceAccount:service-b-XXXXXX-XXXXXX-XXXXXX@gcp-sa-logging.iam.gserviceaccount.com for project "itl-c-logging-9kcl"` returned error: Error applying IAM policy for project "itl-c-logging-9kcl": Error setting IAM policy for project "itl-c-logging-9kcl": googleapi: Error 400: One or more users named in the policy do not belong to a permitted customer.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
    "violations": [
      {
        "description": "User service-b-XXXXXX-XXXXXX-XXXXXX@gcp-sa-logging.iam.gserviceaccount.com is not in permitted organization.",
        "subject": "orgpolicy:projects/itl-c-logging-9kcl?configvalue=service-b-XXXXXX-XXXXXX-XXXXXX%40gcp-sa-logging.iam.gserviceaccount.com",
        "type": "constraints/iam.allowedPolicyMemberDomains"
      }
    ]
  }
]
, failedPrecondition
  with module.logs_export.google_project_iam_member.project_sink_member_billing[0],
  on ../../modules/centralized-logging/main.tf line 220, in resource "google_project_iam_member" "project_sink_member_billing":
 220: resource "google_project_iam_member" "project_sink_member_billing" {

Terraform Configuration

# Must include the domain of the organization you are deploying the foundation.
domains_to_allow = ["itlightning.com"]

essential_contacts_domains_to_allow = ["@itlightning.com"]

scc_notification_name = "scc-notify-iaas"
//scc_notification_filter = "state=\\\"ACTIVE\\\""

remote_state_bucket = "bkt-(redacted)"

data_access_logs_enabled = true

log_export_storage_retention_policy = {
    is_locked             = false
    retention_period_days = 14
}

gcp_groups = {
    audit_viewer = "gcp-audit-viewer@itlightning.com"
    security_reviewer = "gcp-security-reviewer@itlightning.com"
    network_viewer = "gcp-network-viewer@itlightning.com"
    scc_admin = "gcp-scc-admin@itlightning.com"
    global_secrets_admin = "gcp-secrets-admin@itlightning.com"
    kms_admin = "gcp-kms-admin@itlightning.com"
}

# not using hub and spoke networking
enable_hub_and_spoke = false

//create_access_context_manager_access_policy = false

project_budget = {
    dns_hub_budget_amount = 50
    base_net_hub_budget_amount = 50
    base_network_budget_amount = 50
    restricted_net_hub_budget_amount = 50
    restricted_network_budget_amount = 50
    interconnect_budget_amount = 50
    org_secrets_budget_amount = 50
    org_billing_logs_budget_amount = 50
    org_audit_logs_budget_amount = 50
    org_kms_budget_amount = 50
    scc_notifications_budget_amount = 50
}

Terraform Version

In GitHub CI, it's using installing terraform_version: 1.3

Additional information

This was starting from an empty organization before stage 0-bootstrap was run. Any ideas?

[daniel-cit]

Hi @klondikedragon thanks for your report.

The first time 1-org was executed, the configuration of the Organization Policies was executed and the Domain Restrict Sharing one was applied.

This Org Policy prevents user from outside the configured organization to be added to IAM policies.

Your problem is similar to this one in this troubleshooting instructions: BigQuery log sink for a billing account

Could you try to follow these instructions to fix your problem?

[klondikedragon]

@daniel-cit - that was perfect advice thank you! I set the domain sharing restriction policy to allow all, reapplied, and everything went through. It wasn't obvious to me that the log sink for a billing account is considered external. Perhaps a tip about this could be added to the troubleshooting doc? Thanks!

[eeaton]

Glad to hear that this could resolve the issue. We have some backlog tasks to improve troubleshooting for all the edge cases where terraform fails during the apply stage and gets stuck in a weird state. We're also trying to escalate the root issue that some GCP services are not compatible with the domain restricted sharing org policy.

I'll close this issue for now but feel free to re-open if needed.