Open obriensystems opened 1 year ago
Summary: the unsupported argument error - is an expected result of an API change Fix is usually to adapt the module or inherit the variable - here is is just to remove the reference in projects.tf
admin_@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/1-guardrails (ggz-seed-project)$ terraform apply -var-file variables.tfvar
╷
│ Error: Unsupported argument
│
│ on projects.tf line 5, in module "administration":
│ 5: impersonate_service_account = var.terraform_service_account
│
│ An argument named "impersonate_service_account" is not expected here.
https://github.com/canada-ca/accelerators_accelerateurs-gcp/pull/49
Validation
https://github.com/canada-ca/cloud-guardrails-gcp/blob/main/guardrails-validation/README.md
Do you want to continue (Y/n)? ^C
Command killed by keyboard interrupt
admin_root@cloudshell:~$ gcloud config set project validator-ncinfo
admin_root@cloudshell:~ (validator-ncinfo)$ mkdir validator
admin_root@cloudshell:~ (validator-ncinfo)$ cd validator/
admin_root@cloudshell:~/validator (validator-ncinfo)$ export MY_BUCKET_NAME=gr-validator-ncinfo
set billing
admin_root@cloudshell:~/validator (validator-ncinfo)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://gr-validator-ncinfo/...
admin_root@cloudshell:~/validator (validator-ncinfo)$ gcloud asset export --output-path=gs://$MY_BUCKET_NAME/resource_inventory.json --content-type=resource --project=guardrails-eaba
ERROR: (gcloud.asset.export) code: 403
message: service-65453940734@gcp-sa-cloudasset.iam.gserviceaccount.com does not have
storage.objects.create access to the Google Cloud Storage object.
status: PERMISSION_DENIED
admin_root@cloudshell:~/validator (validator-ncinfo)$ gcloud projects add-iam-policy-binding validator-nfinfo --member user:admin-root@cloud-nuage.info --role roles/cloudasset.viewer
ERROR: (gcloud.projects.add-iam-policy-binding) User [admin-root@nuage-cloud.info] does not have permission to access projects instance [validator-nfinfo:getIamPolicy] (or it may not exist): The caller does not have permission
Billing Summary
see (needs update for shared billing) https://cloud.google.com/billing/docs/how-to/billing-access
State of billing id associations for type 2 are the following - notice that the terraform service account is in the list as well as the user sa
billing account administrator
billing account user
logs configuration writer (via tf)
guardrails gcp zone
Reproduced issue where a 3rd party billing id in org 2 is used by org 1 - by adding org 1 sa as BAA in org 2. What happens is the normally inherited roles (beyond BAA) are not properly set in billing when the IAM role is set. As a result Billing Account User must be set manually for the terraform SA in the billing view or we get the following
terraform destroy - the deployments, then change the billing id, terraform init and apply
google_organization_iam_audit_config.org_config[0]: Creation complete after 17s [id=93413315325/audit_config/allServices]
╷
│ Error: failed pre-requisites: missing permission on "billingAccounts/019283-6F1AB5-7AD576": billing.resourceAssociations.create
│
│ with module.administration.module.project-factory.google_project.main,
│ on .terraform/modules/administration/modules/core_project_factory/main.tf line 65, in resource "google_project" "main":
│ 65: resource "google_project" "main" {
│
╵
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/1-guardrails (sscncinfo-seed-project)$ terraform apply -var-file variables.tfvar
manually set the role on both IAM and billing (why?)
If the billing account is from another org - the role will not get inherited automatically from IAM - you must also set it in billing
rerun
Plan: 30 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.org-policy.google_organization_policy.org_policy_list_allow_values[0]: Modifying... [id=93413315325/constraints/gcp.resourceLocations]
module.administration.module.project-factory.google_project.main: Creating...
module.org-policy.google_organization_policy.org_policy_list_allow_values[0]: Modifications complete after 1s [id=93413315325/constraints/gcp.resourceLocations]
module.administration.module.project-factory.google_project.main: Still creating... [10s elapsed]
module.pubsub_destination.google_pubsub_topic_iam_member.pubsub_sink_member: Creation complete after 9s [id=projects/guardrails-5def/topics/tp-org-logs-zvw4/roles/pubsub.publisher/serviceAccount:o93413315325-642767@gcp-sa-logging.iam.gserviceaccount.com]
Apply complete! Resources: 30 added, 1 changed, 0 destroyed.
TL;DR; Shared billing accounts do not get shared IAM roles - they need to be set separately
Keep in mind that we need a workaround (see #47) for the fact that if the billing account is of type "shared" where it comes in under "No Organization, ID=0" then any service account created will not get inherited links from IAM set in Billing - these like Billing Account User - need to be set manually.
Example
michael@cloudshell:~$ gcloud config set project gcp-zone-landing-stg
Updated property [core/project].
michael@cloudshell:~ (gcp-zone-landing-stg)$ export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
michael@cloudshell:~ (gcp-zone-landing-stg)$ export ORG_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
michael@cloudshell:~ (gcp-zone-landing-stg)$ export SA_PREFIX=tfsa-example
michael@cloudshell:~ (gcp-zone-landing-stg)$ gcloud iam service-accounts create "${SA_PREFIX}" --display-name "Terraform example service account" --project=${PROJECT_ID}
Created service account [tfsa-example].
michael@cloudshell:~ (gcp-zone-landing-stg)$ export SA_EMAIL=`gcloud iam service-accounts list --project="${PROJECT_ID}" --filter=tfsa --format="value(email)"`
michael@cloudshell:~ (gcp-zone-landing-stg)$ echo $SA_EMAIL
tfsa-example@gcp-zone-landing-stg.iam.gserviceaccount.com
check existing roles
michael@cloudshell:~ (gcp-zone-landing-stg)$ gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$SA_EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
Set the billing role
gcloud organizations add-iam-policy-binding ${ORG_ID} --member=serviceAccount:${SA_EMAIL} --role=roles/billing.user
check again
michael@cloudshell:~ (gcp-zone-landing-stg)$ gcloud organizations add-iam-policy-binding ${ORG_ID} --member=serviceAccount:${SA_EMAIL} --role=roles/billing.user
Updated IAM policy for organization [925207728429].
...
michael@cloudshell:~ (gcp-zone-landing-stg)$ gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$SA_EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
ROLE: roles/billing.user
It may take a couple min to show in IAM
Checking billing on the shared account
expected on billing accounts belonging to this org - via IAM inheritance in billing
not expected on billing accounts shared from other orgs
Notes on rerunning the script after any type of inadvertent typo in variables.tfvar
You will need to get back into the project (with associated re-auth that will kick in), and cd into the 1- directory before running init and apply again
gcloud config set project gr-glc
cd deployment-templates/Terraform/guardrails/
cd ../1-guardrails/
terraform init
terraform apply -var-file variables.tfvar
Example install https://github.com/canada-ca/accelerators_accelerateurs-gcp/issues/47
Guardrails install instructions We will need the following artifacts at the meeting to proceed. (essentially admin level - domain and BID access) 1 - a BID (Billing ID) - the shared billing account Note: this BID will need at least 3 open project/billing association quota left - new accounts get 5 default - not normally an issue 2 - the GCP org owning the BID above - where we will add the new SA as a BAA (billing account administrator) 3 - any Billing Account Administrator cloud identity user on the GCP domain that owns the BID above - to be able to add the new cloud identity account (Admin: super admin, GCP: organization administrator) that bootstraps the new HC org we will need the account below to be added as a BAA on the BID org 4 - a person with any email that will be the new bootstrap cloud identity super admin (root) account on the new HC org - this user will need to configure MFA (defaulting to a phone text) example: super-admin@guardrails.gcp.zone 5 - Pick a domain name for the GCP org - note that this is just an org identifier not a future FQDN frontend for later - it does not have to be used for A and CNAME records later - something like guardrails.gcp.zone 6 - Domain validation: an IT/OPS person or anyone who can add a TXT domain record to the root zone (synchronously - in the meeting) - GCP org onboarding takes 2 min after the TXT record DNS propagates - the record can be removed after we are done - or left example: guardrails.gcp.zone TXT record on the sub domain gcp.zone
References:
Determine cloud profile 1=sandbox: https://github.com/canada-ca/cloud-guardrails/blob/master/EN/00_Applicable-Scope.md#applicability-of-guardrails-to-cloud-usage-profiles Shared Billing structure: https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#billing Identity Onboarding steps: https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#onboarding-category-3b1-3rd-party-email-account---3rd-party-aws-route53-domain-validation---reuse-existing-billing-account Example Guardrails install run: https://github.com/canada-ca/accelerators_accelerateurs-gcp/issues/47 Guardrails entrypoint to clone code: https://github.com/canada-ca/accelerators_accelerateurs-gcp#gc-accelerators-gcp Guardrails instructions we will follow: https://github.com/canada-ca/accelerators_accelerateurs-gcp/blob/main/deployment-templates/Terraform/guardrails/README.md#setting-up-your-environment
admin at guardrails.gcp.zone clean identity org from scratch - full onboarding prep
note the "trust" checkbox - critical - https://github.com/canada-ca/accelerators_accelerateurs-gcp/issues/41
<img width="1767" alt="Screen Shot 2022-09-15 at 19 08 43" src="https://user-images.githubusercontent.com/24765473/190526445-9e2aa520-a894-4223-a5df-685be
9fa0f25.png">
![Uploading Screen Shot 2022-09-15 at 19.12.16.png…]()
Notice that there is a pending jira I forgot about with the checkout in cloud shell on my branch
before
as expected - we will add missing roles - these will be automated tomorrow+ in https://github.com/canada-ca/accelerators_accelerateurs-gcp/issues/42
create service account token creator at the org level
add org policy admin
add project creator role
add billing project manager role
full bootstrap
check the 0 items returns and change the output from the iam role additions to not print out
after
diff
variables.tfvar
terraform init - initializes provider plugins - verify this
apply
above worked in last admin-root@cloud-nuage.info in mid June 2022 Since then there may have been terraform changes to module, less likely code changes - verifying both before tomorrows's 2 guardrails installs
triage
https://registry.terraform.io/modules/terraform-google-modules/project-factory/google/latest
after 5 min check
asset inventory