Demo: Script: Automated minimal landing zone with a (hub-env and core-landing-zone, client-setup, client-landing-zone, client-project-setup) GitOps/OCI based deployment on a clean GCP organization - walkthrough

[fmichaelobrien]

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps

Base architecture with GCP native firewalls

Cluster with 4 packages clz + 4 client

Extended version with Fortigate NGFW

including projects/hub-env package

graph LR;
    style LZV2 fill:#44f,stroke:#f66,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
    %% mapped and documented
    project/hub-env-->core-landing-zone;
    client-setup;
    client-setup-->dns-project;
    client-setup-->kcc-management-project;
    client-landing-zone-->client-setup;
    client-project-setup-->client-landing-zone;
    client-project-setup-->client-management-project;
    gatekeeper-policies;

    kcc-management-project;
    core-landing-zone-->kcc-management-project;
    dns-project-->core-landing-zone;
    logging-project-->core-landing-zone;
    client-management-project-->client-setup;
    host-project-->client-landing-zone;

mermaid - diagrams as code

Issues

• 20240220: investigate policy based routing for the hub-env package VPC - https://cloud.google.com/vpc/docs/policy-based-routes
• • refer to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494
• https://github.com/fortinet/fortigate-tutorial-gcp/issues/5

• 799

• 801

• 802

• 823

• 824

• https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/842

• 847

Infrastructure

• admin-root nuage-cloud or = cloud foundation checklist
• root landing.systems = hub-env 446 (cluster deleted - infra up including 3 fortigate VMs in hub-env)
• mic obrienlabs.eng = manual lz coding
• mic cloud-setup or = automated lz coding - 4 core/client packages up
• mic obr.ind - previous hub-env, lz only - (cluster deleted 20240217) manual
• branch https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh766-script
• wiki https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps
• review alternative approach using https://cloud.google.com/blog/products/identity-security/introducing-google-cloud-firewall-plus-with-intrusion-prevention

Follow docs starting at https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#3-deploy-the-infrastructure-using-gitops Follow wiki - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#landing-zone-install Follow triage - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#workaroundstriage

Update https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#monorepo-lz Follow monorepo instructions from SSC around the hydration script (wraps kpt fn render)

• https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Landing%20Zone%20Operations/Building.md
• Tier1 https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Landing%20Zone%20Operations/Changing.md#step-3---hydrate
• Tier2 https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Onboarding/Client.md
• Tier34 https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Onboarding/Application.md

This is the OCI (repo tracking) version of deployment - not the minimal KPT based deployment in #611

Architecture

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Architecture

Inventory

Package: core-landing-zone

Package: client-setup

Package: client-landing-zone

Package: client-project-setup

Package: projects/hub-env

• IaaS security - not serverless Firewall Plus

Package: gatekeeper-policies

Minimal Landing Zone from a clean GCP organization using a single script - use for development or CI/CD

FinOps: PAYG + GKE + GCE costs will be $80/day above the normal $10/day for the GKE cluster alone.

This jira will document standing up a subset of the full landing zone consisting of the following 2 packages in a clean org References

• 446

• 258

• repo diff https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/612

See ongoing documentation in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#landing-zone-user-procedures

• see https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt#delete_the_resources
• see https://cloud.google.com/config-connector/docs/how-to/managing-deleting-resources
• see https://cloud.google.com/anthos-config-management/docs/tutorials/manage-resources-config-controller#delete_the_individual_resources

• Clean Organization

• see onboarding at https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Onboarding

  Clean super admin

The super admin account will have the organization administrator role and be able to create projects through the project creator role on the organization (all users inherit this and we will lock it down later)

Launch shell.cloud.google.com

navigate to https://shell.cloud.google.com

By default the user is an Organization Administrator We will add required roles to get the point of creating a bootstrap project and then let the landing zone setup script take over adding roles required for LZ bootstrap.

For those customers on direct billing - activate your credits

5 billing projects required

Prepare for increasing billing quota above 5 projects by paying early $50 and asking for a billing quota increase 2 days later - for how use shared billing to go past 5. For the purposes of the core-landing-zone and hub-env you need 1 bootstrap project, 1 config controller project, a logging and dns project and a hub project. Therefore disable billing on "My first project" to have all 5 for now.

follow instructions to increase your billing account quota to above 10 (I asked for 10 in addition to the default 5) using our instructions below

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Onboarding#billing-quota fill out https://support.google.com/code/contact/billing_quota_increase

Usually you are approved within 60 seconds

license key config https://github.com/fortinetsolutions/terraform-modules/blob/master/GCP/modules/fortigate_byol/main.tf#L33

see also https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt

[obriensystems]

docs adjust

• update staging docs in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#monorepo-lz

[obriensystems]

walkthrough triage and issues spawn

Follow monorepo instructions from SSC around the hydration script (wraps kpt fn render)

• https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Landing%20Zone%20Operations/Building.md
• Tier1 https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Landing%20Zone%20Operations/Changing.md#step-3---hydrate
• Tier2 https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Onboarding/Client.md
• Tier34 https://github.com/ssc-spc-ccoe-cei/gcp-documentation/blob/main/Onboarding/Application.md

[fmichaelobrien]

move the partially completed kpt version script in 446 that completed the core-landing-zone and was mid way through hub-env

446

• Add client-landing-zone and client-setup
• merge periodic package automations into main in phases

[obriensystems]

New org adjustment - case Route53 domain, GCP Workspace admin Email DNS verification - don't use @ as mx record id - leave blank

[obriensystems]

Clean org 2 - cloud-setup.org

• project quota is 12
• billing quota is 5 -

Add IAM permissions to be able to create a project at the org or folder level

We need this before we can clone the script and continue from there

project billing manager (to associate billing)
folder creator/admin for resourcemanager.folders.create
project creator/deleter for resourcemanager.projects.create

Create folder

 gcloud resource-manager folders create --display-name=lz20240127 --organization=$ORG_ID

Create bootstrap project - default at the org level

 gcloud projects create kcc-cso --name="kcc-cso" --set-as-default 

Associate billing

gcloud beta billing projects link kcc-cso --billing-account $BILLING_ID

[obriensystems]

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit (kcc-cso)$ cd solutions/
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ ./setup.sh -b kcc-cso -u cso -n true -c true -l false -h false -r false -d false -j false
existing project: 
Date: Sat 27 Jan 2024 01:22:31 PM UTC
Timestamp: 1706361751
running with: -b kcc-cso -u cso -c true -l false -h false -r false -d false -p 
Updated property [core/project].
Switched back to boot project kcc-cso
Start: 1706361752
unique string: cso
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Creating project: kcc-cso-4380
CC_PROJECT_ID: kcc-cso-4380
BOOT_PROJECT_ID: kcc-cso
BILLING_ID: 01B35D-D56E1A-BAE17A
ORG_ID: 734065690346
applying roles to the super admin SUPER_ADMIN_EMAIL: michael@cloud-setup.org
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Creating KCC project: kcc-cso-4380 on folder: 276061734969
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-cso-4380].
Waiting for [operations/cp.6824382625317520983] to finish...done.                                                                                                                             
Enabling service [cloudapis.googleapis.com] on project [kcc-cso-4380]...
Operation "operations/acat.p2-343139601407-b95e383a-72f5-44ff-a449-41414d16cc51" finished successfully.
Updated property [core/project] to [kcc-cso-4380].
Updated property [core/project].
Enabling billing on account: 01B35D-D56E1A-BAE17A
billingAccountName: billingAccounts/01B35D-D56E1A-BAE17A
billingEnabled: true
name: projects/kcc-cso-4380/billingInfo
projectId: kcc-cso-4380
sleep 45 sec before enabling services

peration "operations/acf.p2-343139601407-8657e213-5fad-4e8b-b6d6-289dcb3e7caf" finished successfully.
Operation "operations/acat.p2-343139601407-2090c0f9-5a02-430f-ab57-18c38528d39d" finished successfully.
Operation "operations/acat.p2-343139601407-e00a3300-8855-4169-8384-89d01eb67585" finished successfully.
Operation "operations/acat.p2-343139601407-cc25ed96-5599-4f56-a00f-49ea572af104" finished successfully.
Operation "operations/acat.p2-343139601407-6bf08c04-e664-4aed-b0aa-32c632052d4f" finished successfully.
Operation "operations/acat.p2-343139601407-b4f6e83b-533a-406e-b4a1-4a2445b5e0be" finished successfully.
name: organizations/734065690346/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc

peration "operations/acf.p2-343139601407-8657e213-5fad-4e8b-b6d6-289dcb3e7caf" finished successfully.
Operation "operations/acat.p2-343139601407-2090c0f9-5a02-430f-ab57-18c38528d39d" finished successfully.
Operation "operations/acat.p2-343139601407-e00a3300-8855-4169-8384-89d01eb67585" finished successfully.
Operation "operations/acat.p2-343139601407-cc25ed96-5599-4f56-a00f-49ea572af104" finished successfully.
Operation "operations/acat.p2-343139601407-6bf08c04-e664-4aed-b0aa-32c632052d4f" finished successfully.
Operation "operations/acat.p2-343139601407-b4f6e83b-533a-406e-b4a1-4a2445b5e0be" finished successfully.
name: organizations/734065690346/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc

Created [https://www.googleapis.com/compute/v1/projects/kcc-cso-4380/global/networks/kcc-ls-vpc].
NAME: kcc-ls-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp:22,tcp:3389,icmp

Create subnet kcc-ls-sn off VPC: kcc-ls-vpc using 192.168.0.0/16 on region: northamerica-northeast1
Created [https://www.googleapis.com/compute/v1/projects/kcc-cso-4380/regions/northamerica-northeast1/subnetworks/kcc-ls-sn].
NAME: kcc-ls-sn
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
create default firewalls
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc on project kcc-cso-4380
Create request issued for: [kcc]

Waiting for operation [projects/kcc-cso-4380/locations/northamerica-northeast1/operations/operation-1706361984166-60fed5becaa35-a8a104b7-388f87a2] to complete...working..                    
Waiting for operation [projects/kcc-cso-4380/locations/northamerica-northeast1/operations/operation-1706361984166-60fed5becaa35-a8a104b7-388f87a2] to complete...working...                   
Waiting for operation [projects/kcc-cso-4380/locations/northamerica-northeast1/operations/operation-1706361984166-60fed5becaa35-a8a104b7-388f87a2] to complete...done.                        
Created instance [kcc].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
Cluster create time: 1106 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
List Clusters:
NAME: kcc
LOCATION: northamerica-northeast1
STATE: RUNNING
post GKE cluster create - applying 2 roles to org: 734065690346 and project: kcc-cso-4380 on the yakima gke service account to prep for kpt deployment: service-343139601407@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [734065690346].
Updated IAM policy for project [kcc-cso-4380].
Updated IAM policy for organization [734065690346].
Updated IAM policy for organization [734065690346].
Total Duration: 1350 sec
Date: Sat 27 Jan 2024 01:45:02 PM UTC
Timestamp: 1706363102
Updated property [core/project].
Switched back to boot project kcc-cso
**** Done ****
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ 

[obriensystems]

deploy lz

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions$ gcloud config set project kcc-cso
Updated property [core/project].
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ ./setup.sh -b kcc-cso -u cso -n false -c false -l true -h false -r false -d false -j false -p kcc-cso-4380

existing project: kcc-cso-4380
Date: Tue 30 Jan 2024 06:36:58 PM UTC
Timestamp: 1706639818
running with: -b kcc-cso -u cso -c false -l true -h false -r false -d false -p kcc-cso-4380
Updated property [core/project].
Switched back to boot project kcc-cso
Start: 1706639820
unique string: cso
REGION: northamerica-northeast1
NETWORK: kcc-vpc
SUBNET: kcc-sn
CLUSTER: kcc
Reusing project: kcc-cso-4380
CC_PROJECT_ID: kcc-cso-4380
BOOT_PROJECT_ID: kcc-cso
BILLING_ID: 01B35D-D56E1A-BAE17A
ORG_ID: 734065690346
Switching to KCC project kcc-cso-4380
Updated property [core/project].
wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER: 343139601407
DIRECTORY_CUSTOMER_ID: C02w06bdi
generated derived setters-core-landing-zone.yaml
Creating kpt
deploying core-landing-zone
get kpt release package solutions/core-landing-zone version 0.7.1
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.7.1 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso1/locations/northamerica-northeast1/buckets/security-log-bucket"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-cso-4380-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-accesscontextmanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-anthos apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudbilling apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudresourcemanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-serviceusage apply successful
apply phase finished
reconcile phase started
namespace/hierarchy reconcile successful
namespace/logging reconcile successful
namespace/networking reconcile successful
namespace/policies reconcile successful
namespace/projects reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-cso-4380-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-accesscontextmanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-anthos reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudbilling reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudresourcemanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-serviceusage reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile failed
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-anthos reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-accesscontextmanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudbilling reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-serviceusage reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudresourcemanager reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile successful

iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-cso-4380-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful

iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile successful
reconcile phase finished
apply phase started

rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions apply successful
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380 apply successful
folder.resourcemanager.cnrm.cloud.google.com/audits apply successful
folder.resourcemanager.cnrm.cloud.google.com/clients apply successful
folder.resourcemanager.cnrm.cloud.google.com/services apply successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure apply successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380 apply successful

esourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 apply successful
apply phase finished
reconcile phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions reconcile pending
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380 reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/services reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/services reconcile failed
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380 reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services reconcile successful

folder.resourciampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380 reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380 reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 reconcile successful

2min

iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions reconcile successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1 reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions reconcile successful
reconcile phase finished
apply phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions apply successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 apply successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket apply successful
project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1 apply successful
service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-logging apply successful
service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-monitoring apply successful
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket apply successful
apply phase finished
reconcile phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket reconcile pending
project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1 reconcile pending
service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-logging reconcile pending
service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-monitoring reconcile pending
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket reconcile pending
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket reconcile successful
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 reconcile successful

1344 projects coming in - logging

1348
project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1 reconcile successful

1359: terminate 437 kpt live apply

run

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$   kubens config-control
Context "gke_kcc-cso-4380_northamerica-northeast1_krmapihost-kcc" modified.
Active namespace is "config-control".
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp | grep UpdateFailed | wc -l
0
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp 
NAME                                                                     AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa   25m   True    UpToDate   25m
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa          25m   True    UpToDate   25m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa                 25m   True    UpToDate   25m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa                   25m   True    UpToDate   25m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa                25m   True    UpToDate   25m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa                  25m   True    UpToDate   25m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa                  25m   True    UpToDate   25m

NAME                                                                                              AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding   25m   True    UpToDate   25m
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding          25m   True    UpToDate   25m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding                 25m   True    UpToDate   25m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding                   25m   True    UpToDate   25m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding                25m   True    UpToDate   25m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding                  25m   True    UpToDate   25m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding                  25m   True    UpToDate   25m

NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           25m   True    UpToDate   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   25m   True    UpToDate   25m

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions                   25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   25m   True    UpToDate   24m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-cso-4380-permissions                   25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          25m   True    UpToDate   24m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 25m   True    UpToDate   24m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 25m   True    UpToDate   24m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                25m   True    UpToDate   24m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   25m   True    UpToDate   24m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              25m   True    UpToDate   24m

NAME                                                                           AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-accesscontextmanager   25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-anthos                 25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudbilling           25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudresourcemanager   25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-serviceusage           25m   True    UpToDate   25m

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n projects
NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   19m   False   DependencyNotFound   19m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         19m   False   DependencyNotFound   19m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               19m   False   DependencyNotFound   19m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      19m   False   DependencyNotFound   19m

NAME                                                                                                     AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions   25m   True    UpToDate   22m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions       25m   True    UpToDate   19m

NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   19m   True    UpToDate   19m

NAME                                                                 AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1       19m   True    UpToDate   15m
project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1   25m   True    UpToDate   19m

NAME                                                                         AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-logging      19m   True    UpToDate   19m
service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-monitoring   19m   True    UpToDate   19m
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n networking
No resources found in networking namespace.
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    26m   True    UpToDate   25m
folder.resourcemanager.cnrm.cloud.google.com/clients                   26m   True    UpToDate   25m
folder.resourcemanager.cnrm.cloud.google.com/services                  26m   True    UpToDate   25m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   26m   True    UpToDate   25m
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n policies
NAME                                                                                                                  AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380   26m   True    UpToDate   24m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380           26m   True    UpToDate   24m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380      26m   True    UpToDate   24m
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n logging
NAME                                                                                    AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1   20m   True    UpToDate   20m
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                      20m   True    UpToDate   20m

NAME                                                                       AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380   26m   True    UpToDate   20m

NAME                                                                       AGE   READY   STATUS     STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket   20m   True    UpToDate   20m

Issues with missing networking namespace artifacts and permissions on projects

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n projects
NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   19m   False   DependencyNotFound   19m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         19m   False   DependencyNotFound   19m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               19m   False   DependencyNotFound   19m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      19m   False   DependencyNotFound   19m

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions -n projects

  Warning  DependencyNotFound  4m15s (x3 over 23m)  iampartialpolicy-controller  reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found

mgmt-project/project-sink.yaml
spec:
  projectRef:
    external: kcc-cso-4380 # kpt-set: ${management-project-id}
  destination:
    # AU-3, AU-3(1), AU-4(1), AU-6(4), AU-9(2)
    loggingLogBucketRef:
      external: logging.googleapis.com/projects/logging-project-cso1/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-cso1 # kpt-set: logging.googleapis.com/projects/${logging-project-id}/locations/northamerica-northeast1/buckets/${platform-and-component-log-bucket}

checking 
looks like the name is not rendered correctly - missing cso1
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 apply successful
from setters.yaml
  platform-and-component-log-bucket: platform-and-component-log-bucket-cso1

check sa
kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa
  Normal  UpToDate  40m (x2 over 40m)  iamserviceaccount-controller  The resource is up to date

don't need to do
gcloud beta billing accounts add-iam-policy-binding "${BILLING_ID}" --member "serviceAccount:projects-sa@${KCC_PROJECT_ID}.iam.gserviceaccount.com" --role "roles/billing.user"

DNS managed zone is missing because of service permission on dns.googleapis.com

[obriensystems]

triage skipped resources on pass 1

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status core-landing-zone
inventory-49821483/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-49821483/logginglogbucket.logging.cnrm.cloud.google.com/logging/security-log-bucket is Current: Resource is Current
inventory-49821483/logginglogbucket.logging.cnrm.cloud.google.com/logging/platform-and-component-log-bucket-cso1 is Current: Resource is Current
inventory-49821483/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Current: Resource is Current
inventory-49821483/monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/logging/kcc-cso-4380 is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/org-log-sink-security-logging-project-cso1 is not found
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-log-sink is not found
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-infra-log-sink is not found
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
inventory-49821483/iamauditconfig.iam.cnrm.cloud.google.com/projects/logging-project-data-access-log-config is Current: Resource is Current
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-cso1-data-access-sink is NotFound: Resource not found
inventory-49821483/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-cso1 is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-cso1-logging is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-cso1-monitoring is Current: Resource is Current
inventory-49821483/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is NotFound: Resource not found
inventory-49821483/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-49821483/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-cso1-standard-core-public-dns is NotFound: Resource not found
inventory-49821483/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-cso1 is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-cso1-dns is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is NotFound: Resource not found
inventory-49821483/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-49821483/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging-except-kcc-cso-4380 is Current: Resource is Current
inventory-49821483/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-kcc-cso-4380 is Current: Resource is Current
inventory-49821483/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 is Current: Resource is Current
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is NotFound: Resource not found
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-cloudbilling is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-cloudresourcemanager is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-serviceusage is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-accesscontextmanager is Current: Resource is Current
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-anthos is Current: Resource is Current
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-metric-writer-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/config-management-monitoring/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/namespace//hierarchy is Current: Resource is current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-49821483/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-49821483/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-49821483/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-monitoring-admin-kcc-cso-4380-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-monitoring-admin-logging-project-cso1-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-storageadmin-logging-project-cso1-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/namespace//logging is Current: Resource is current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/namespace//networking is Current: Resource is current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/namespace//policies is Current: Resource is current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-49821483/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-49821483/namespace//projects is Current: Resource is current
inventory-49821483/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-49821483/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-49821483/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-49821483/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-subnetwork-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-49821483/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier4-secretmanager-admin is Current: Resource is Current
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-cso1 is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-cso1 is NotFound: Resource not found

missing 12

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status core-landing-zone | grep not 
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/org-log-sink-security-logging-project-cso1 is not found
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-log-sink is not found
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-infra-log-sink is not found
inventory-49821483/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-cso1-data-access-sink is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is NotFound: Resource not found
inventory-49821483/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-cso1-standard-core-public-dns is NotFound: Resource not found
inventory-49821483/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-cso1-dns is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-cso1 is NotFound: Resource not found
inventory-49821483/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-cso1 is NotFound: Resource not found

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                      READY   STATUS    RESTARTS        AGE
cnrm-system                       cnrm-controller-manager-2ngn5mteag2v6r5itiwa-0            2/2     Running   0               56m
cnrm-system                       cnrm-controller-manager-c3w3isgmjny4adkmvixa-0            2/2     Running   0               54m
cnrm-system                       cnrm-controller-manager-it3zylhy24y5aobjjbha-0            2/2     Running   0               54m
cnrm-system                       cnrm-controller-manager-jl4awbbvx5nutfp7yq3a-0            2/2     Running   0               56m
cnrm-system                       cnrm-controller-manager-ovcmntvmtm3wq73uhuzq-0            2/2     Running   0               54m
cnrm-system                       cnrm-controller-manager-p2jcfga4lzvznyzcjuha-0            2/2     Running   0               4h23m
cnrm-system                       cnrm-controller-manager-yfi5fm3zvyuoan6qjobq-0            2/2     Running   0               54m
cnrm-system                       cnrm-controller-manager-zw3egolzoau5iyevttxa-0            2/2     Running   0               54m
cnrm-system                       cnrm-deletiondefender-0                                   1/1     Running   0               4h16m
cnrm-system                       cnrm-resource-stats-recorder-6b78d6845b-b5mdr             2/2     Running   0               4h19m
cnrm-system                       cnrm-unmanaged-detector-0                                 1/1     Running   0               4h19m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-75z5c                     1/1     Running   0               4h16m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-bqfzt                     1/1     Running   0               52m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-d952k                     1/1     Running   0               55m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-jp7sp                     1/1     Running   0               4h19m
config-management-monitoring      otel-collector-78d4f5dd58-snjrd                           1/1     Running   0               4h16m
config-management-system          config-management-operator-684f5b9d88-swkbs               1/1     Running   0               4h16m
config-management-system          reconciler-manager-586bb6cdd6-rhn7t                       2/2     Running   0               4h16m
configconnector-operator-system   configconnector-operator-0                                2/2     Running   0               4h23m
configsync-healthcheck-system     configsync-healthcheck-service-84c45b9679-h92wp           1/1     Running   0               4h23m
gatekeeper-system                 gatekeeper-audit-6d968dfb8d-dp2rt                         1/1     Running   0               4h19m
gatekeeper-system                 gatekeeper-controller-manager-6ffb74c54-ppf7b             1/1     Running   0               4h19m
gke-gmp-system                    alertmanager-0                                            2/2     Running   2 (4h19m ago)   4h19m
gke-gmp-system                    collector-8m79f                                           2/2     Running   1 (52m ago)     54m
gke-gmp-system                    collector-p6vrg                                           2/2     Running   1 (4h15m ago)   4h17m
gke-gmp-system                    collector-qszdz                                           2/2     Running   1 (4h19m ago)   4h20m
gke-gmp-system                    collector-x7kxd                                           2/2     Running   1 (4h22m ago)   4h24m
gke-gmp-system                    gmp-operator-6555c6d658-8ppdw                             1/1     Running   0               4h19m
gke-gmp-system                    rule-evaluator-554bf88db9-7fxzv                           2/2     Running   2 (4h19m ago)   4h19m
krmapihosting-monitoring          krmapihosting-metrics-agent-dnx6h                         1/1     Running   0               4h20m
krmapihosting-monitoring          krmapihosting-metrics-agent-jnklk                         1/1     Running   0               4h16m
krmapihosting-monitoring          krmapihosting-metrics-agent-mf5ml                         1/1     Running   0               53m
krmapihosting-monitoring          krmapihosting-metrics-agent-w85wl                         1/1     Running   0               4h23m
krmapihosting-system              bootstrap-57495bff9c-4trsm                                1/1     Running   0               4h19m
kube-system                       anetd-4dxhz                                               1/1     Running   0               4h24m
kube-system                       anetd-bkmm4                                               1/1     Running   0               53m
kube-system                       anetd-f4ngq                                               1/1     Running   0               4h17m
kube-system                       anetd-s6gn7                                               1/1     Running   0               4h20m
kube-system                       antrea-controller-horizontal-autoscaler-67df5fcf9-mm2jv   1/1     Running   0               4h19m
kube-system                       egress-nat-controller-5bc948f77b-p9wzs                    1/1     Running   0               4h16m
kube-system                       event-exporter-gke-5b8bcb44f7-hgg6p                       2/2     Running   0               4h16m
kube-system                       filestore-node-4htjn                                      3/3     Running   0               4h17m
kube-system                       filestore-node-h2g7q                                      3/3     Running   0               4h24m
kube-system                       filestore-node-kzzkr                                      3/3     Running   0               54m
kube-system                       filestore-node-q5sbn                                      3/3     Running   0               4h20m
kube-system                       fluentbit-gke-big-9mh66                                   2/2     Running   0               4h20m
kube-system                       fluentbit-gke-big-mrs2k                                   2/2     Running   0               4h24m
kube-system                       fluentbit-gke-big-vsm6l                                   2/2     Running   0               4h17m
kube-system                       fluentbit-gke-big-zg82m                                   2/2     Running   0               53m
kube-system                       gcsfusecsi-node-69x7m                                     2/2     Running   0               4h20m
kube-system                       gcsfusecsi-node-898x8                                     2/2     Running   0               4h24m
kube-system                       gcsfusecsi-node-plssx                                     2/2     Running   0               4h17m
kube-system                       gcsfusecsi-node-tmw55                                     2/2     Running   0               54m
kube-system                       gke-metadata-server-ddsjl                                 1/1     Running   0               4h20m
kube-system                       gke-metadata-server-nfplr                                 1/1     Running   0               4h24m
kube-system                       gke-metadata-server-tblhg                                 1/1     Running   0               53m
kube-system                       gke-metadata-server-zwjsj                                 1/1     Running   0               4h17m
kube-system                       gke-metrics-agent-4tt9c                                   2/2     Running   0               4h24m
kube-system                       gke-metrics-agent-8b4p2                                   2/2     Running   0               4h17m
kube-system                       gke-metrics-agent-fmknp                                   2/2     Running   0               54m
kube-system                       gke-metrics-agent-gtt6r                                   2/2     Running   0               4h20m
kube-system                       image-package-extractor-64fdb                             1/1     Running   0               4h24m
kube-system                       image-package-extractor-plzxm                             1/1     Running   0               54m
kube-system                       image-package-extractor-tkfxv                             1/1     Running   0               4h20m
kube-system                       image-package-extractor-zqvl9                             1/1     Running   0               4h17m
kube-system                       ip-masq-agent-6cdxt                                       1/1     Running   0               53m
kube-system                       ip-masq-agent-jz56v                                       1/1     Running   0               4h24m
kube-system                       ip-masq-agent-mz5cx                                       1/1     Running   0               4h17m
kube-system                       ip-masq-agent-vmj96                                       1/1     Running   0               4h20m
kube-system                       konnectivity-agent-5b687c8dcb-g7xvx                       1/1     Running   0               4h16m
kube-system                       konnectivity-agent-5b687c8dcb-gr49q                       1/1     Running   0               53m
kube-system                       konnectivity-agent-5b687c8dcb-m2jvs                       1/1     Running   0               4h17m
kube-system                       konnectivity-agent-5b687c8dcb-mzlh5                       1/1     Running   0               4h20m
kube-system                       konnectivity-agent-autoscaler-5d9dbcc6d8-jf97t            1/1     Running   0               4h16m
kube-system                       kube-dns-6f9b8847ff-gvqtq                                 4/4     Running   0               4h16m
kube-system                       kube-dns-6f9b8847ff-kkxcq                                 4/4     Running   0               4h16m
kube-system                       kube-dns-autoscaler-84b8db4dc7-6bz2r                      1/1     Running   0               4h16m
kube-system                       l7-default-backend-cf7cdc6f6-q7hgn                        1/1     Running   0               4h16m
kube-system                       metrics-server-v0.5.2-8fb865474-w29vv                     2/2     Running   0               4h16m
kube-system                       netd-7tcd7                                                1/1     Running   0               4h24m
kube-system                       netd-r2kn2                                                1/1     Running   0               4h20m
kube-system                       netd-s95rf                                                1/1     Running   0               4h17m
kube-system                       netd-trn8z                                                1/1     Running   0               53m
kube-system                       node-local-dns-5jzpx                                      1/1     Running   0               4h20m
kube-system                       node-local-dns-5zdqq                                      1/1     Running   0               4h24m
kube-system                       node-local-dns-b8ccf                                      1/1     Running   0               4h17m
kube-system                       node-local-dns-zfl42                                      1/1     Running   0               53m
kube-system                       pdcsi-node-857k2                                          2/2     Running   0               54m
kube-system                       pdcsi-node-9w7zk                                          2/2     Running   0               4h24m
kube-system                       pdcsi-node-d2x7k                                          2/2     Running   0               4h20m
kube-system                       pdcsi-node-q987f                                          2/2     Running   0               4h17m
resource-group-system             resource-group-controller-manager-66dbd5bdcf-9r22b        2/2     Running   0               4h23m

[obriensystems]

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt pkg tree core-landing-zone
Package "core-landing-zone"
├── [Kptfile]  Kptfile core-landing-zone
├── [resourcegroup.yaml]  ResourceGroup config-control/inventory-49821483
├── [setters.yaml]  ConfigMap setters
├── audits
│   ├── [folder.yaml]  Folder hierarchy/audits
│   └── logging-project
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/platform-and-component-log-bucket-cso1
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/security-log-bucket
│       ├── [cloud-storage-buckets.yaml]  StorageBucket logging/security-incident-log-bucket
│       ├── [project-iam.yaml]  IAMAuditConfig projects/logging-project-data-access-log-config
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-infra-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/security-log-bucket-writer-permissions
│       ├── [project-sink.yaml]  LoggingLogSink logging/logging-project-cso1-data-access-sink
│       ├── [project.yaml]  Project projects/logging-project-cso1
│       ├── [services.yaml]  Service projects/logging-project-cso1-logging
│       ├── [services.yaml]  Service projects/logging-project-cso1-monitoring
│       └── monitoring
│           └── [metrics-scope.yaml]  MonitoringMonitoredProject logging/kcc-cso-4380
├── clients
│   └── [folder.yaml]  Folder hierarchy/clients
├── services
│   ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-log-sink
│   ├── [folder.yaml]  Folder hierarchy/services
│   └── services-infrastructure
│       ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-infra-log-sink
│       ├── [folder.yaml]  Folder hierarchy/services-infrastructure
│       └── dns-project
│           ├── [dns.yaml]  DNSManagedZone networking/dns-project-cso1-standard-core-public-dns
│           ├── [project.yaml]  Project projects/dns-project-cso1
│           └── [services.yaml]  Service projects/dns-project-cso1-dns
├── mgmt-project
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-accesscontextmanager
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-anthos
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-cloudbilling
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-cloudresourcemanager
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-serviceusage
│   └── org-policies
│       ├── [compute-disable-serial-port-logging-except-mgt-project.yaml]  ResourceManagerPolicy policies/compute-disable-serial-port-logging-except-kcc-cso-4380
│       ├── [compute-require-shielded-vm-except-mgmt-project.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm-except-kcc-cso-4380
│       └── [compute-restrict-cloud-nat-usage-except-mgt-project.yaml]  ResourceManagerPolicy policies/compute-restrict-cloud-nat-usage-except-kcc-cso-4380
├── namespaces
│   ├── [config-management-monitoring.yaml]  IAMServiceAccount config-control/config-mgmt-mon-default-sa
│   ├── [config-management-monitoring.yaml]  IAMPolicyMember config-control/config-mgmt-mon-default-sa-metric-writer-permissions
│   ├── [config-management-monitoring.yaml]  IAMPartialPolicy config-control/config-mgmt-mon-default-sa-workload-identity-binding
│   ├── [config-management-monitoring.yaml]  ConfigConnectorContext config-management-monitoring/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [gatekeeper-system.yaml]  IAMServiceAccount config-control/gatekeeper-admin-sa
│   ├── [gatekeeper-system.yaml]  IAMPolicyMember config-control/gatekeeper-admin-sa-metric-writer-permissions
│   ├── [gatekeeper-system.yaml]  IAMPartialPolicy config-control/gatekeeper-admin-sa-workload-identity-binding
│   ├── [gatekeeper-system.yaml]  ConfigConnectorContext gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [hierarchy.yaml]  Namespace hierarchy
│   ├── [hierarchy.yaml]  IAMServiceAccount config-control/hierarchy-sa
│   ├── [hierarchy.yaml]  IAMPolicyMember config-control/hierarchy-sa-folderadmin-permissions
│   ├── [hierarchy.yaml]  IAMPartialPolicy config-control/hierarchy-sa-workload-identity-binding
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-folders-resource-reference-to-logging
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-config-control
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-policies
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-projects
│   ├── [hierarchy.yaml]  ConfigConnectorContext hierarchy/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  Namespace logging
│   ├── [logging.yaml]  IAMServiceAccount config-control/logging-sa
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-logadmin-permissions
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-monitoring-admin-kcc-cso-4380-permissions
│   ├── [logging.yaml]  IAMPartialPolicy config-control/logging-sa-workload-identity-binding
│   ├── [logging.yaml]  RoleBinding logging/allow-logging-resource-reference-from-projects
│   ├── [logging.yaml]  ConfigConnectorContext logging/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  IAMPolicyMember projects/logging-sa-monitoring-admin-logging-project-cso1-permissions
│   ├── [logging.yaml]  IAMPolicyMember projects/logging-sa-storageadmin-logging-project-cso1-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-editor-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-serviceaccountadmin-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-orgroleadmin-permissions
│   ├── [networking.yaml]  Namespace networking
│   ├── [networking.yaml]  IAMServiceAccount config-control/networking-sa
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-dns-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-networkadmin-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-security-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-service-control-org-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-servicedirectoryeditor-permissions
│   ├── [networking.yaml]  IAMPartialPolicy config-control/networking-sa-workload-identity-binding
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-xpnadmin-permissions
│   ├── [networking.yaml]  ConfigConnectorContext networking/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [policies.yaml]  Namespace policies
│   ├── [policies.yaml]  IAMServiceAccount config-control/policies-sa
│   ├── [policies.yaml]  IAMPolicyMember config-control/policies-sa-orgpolicyadmin-permissions
│   ├── [policies.yaml]  IAMPartialPolicy config-control/policies-sa-workload-identity-binding
│   ├── [policies.yaml]  ConfigConnectorContext policies/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [projects.yaml]  Namespace projects
│   ├── [projects.yaml]  IAMServiceAccount config-control/projects-sa
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-billinguser-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectcreator-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectdeleter-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectiamadmin-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectmover-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-serviceusageadmin-permissions
│   ├── [projects.yaml]  IAMPartialPolicy config-control/projects-sa-workload-identity-binding
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-logging
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-networking
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-policies
│   └── [projects.yaml]  ConfigConnectorContext projects/configconnectorcontext.core.cnrm.cloud.google.com
└── org
    ├── [org-sink.yaml]  LoggingLogSink logging/org-log-sink-data-access-logging-project-cso1
    ├── [org-sink.yaml]  LoggingLogSink logging/org-log-sink-security-logging-project-cso1
    └── custom-roles
        ├── [gke-firewall-admin.yaml]  IAMCustomRole config-control/gke-firewall-admin
        ├── [tier2-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier2-dnsrecord-admin
        ├── [tier2-vpcpeering-admin.yaml]  IAMCustomRole config-control/tier2-vpcpeering-admin
        ├── [tier3-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier3-dnsrecord-admin
        ├── [tier3-firewallrule-admin.yaml]  IAMCustomRole config-control/tier3-firewallrule-admin
        ├── [tier3-subnetwork-admin.yaml]  IAMCustomRole config-control/tier3-subnetwork-admin
        ├── [tier3-vpcsc-admin.yaml]  IAMCustomRole config-control/tier3-vpcsc-admin
        └── [tier4-secretmanager-admin.yaml]  IAMCustomRole config-control/tier4-secretmanager-admin

rerunning apply for 12 services spawning https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/799 and docs https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/800

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply core-landing-zone --reconcile-timeout=10m --output=table

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    63m     Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    63m     Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    63m     Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    63m     Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    63m     Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier3-subnetwork-admin      Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMCustomRole/tier4-secretmanager-admin   Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/config-mgmt-mon-default  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/config-mgmt-mon-default-  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/logging-sa-monitoring-ad  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/config-mgmt-mon-defaul  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/hierarchy-sa            Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/logging-sa              Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/networking-sa           Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/policies-sa             Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  IAMServiceAccount/projects-sa             Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  Service/kcc-cso-4380-accesscontextmanage  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  Service/kcc-cso-4380-anthos               Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  Service/kcc-cso-4380-cloudbilling         Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  Service/kcc-cso-4380-cloudresourcemanage  Successful    Current                 Ready                                     63m     Resource is Current                     
config-con  Service/kcc-cso-4380-serviceusage         Successful    Current                 Ready                                     63m     Resource is Current                     
config-man  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    63m     status.healthy is true                  
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    63m     status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    62m     status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    62m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    62m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    62m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    62m     Resource is current                     
hierarchy   Folder/audits                             Successful    Current                 Ready                                     62m     Resource is Current                     
hierarchy   Folder/clients                            Successful    Current                 Ready                                     62m     Resource is Current                     
hierarchy   Folder/services                           Successful    Current                 Ready                                     62m     Resource is Current                     
hierarchy   Folder/services-infrastructure            Successful    Current                 Ready                                     62m     Resource is Current                     
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    62m     status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Successful    Current                 Ready                                     56m     Resource is Current                     
logging     LoggingLogBucket/security-log-bucket      Successful    Current                 Ready                                     56m     Resource is Current                     
logging     LoggingLogSink/logging-project-cso1-data  Pending       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-plat  Pending       Unknown                 -                                         -                                               
logging     LoggingLogSink/org-log-sink-data-access-  Pending       Unknown                 -                                         -                                               
logging     LoggingLogSink/org-log-sink-security-log  Pending       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Pending       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Pending       Unknown                 -                                         -                                               
logging     MonitoringMonitoredProject/kcc-cso-4380   Successful    Current                 Ready                                     62m     Resource is Current                     
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    62m     Resource is current                     
logging     StorageBucket/security-incident-log-buck  Successful    Current                 Ready                                     56m     Resource is Current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    62m     status.healthy is true                  
networking  DNSManagedZone/dns-project-cso1-standard  Pending       Unknown                 -                                         -                                               
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    62m     status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     62m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     62m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-c  Successful    Current                 Ready                                     62m     Resource is Current                     
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    62m     status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Successful    Current                 Ready                                     56m     Resource is Current                     
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Successful    InProgress              Ready                                     56m     reference LoggingLogSink logging/mgmt-pr
projects    IAMPartialPolicy/platform-and-component-  Successful    InProgress              Ready                                     56m     reference LoggingLogSink logging/platfor
projects    IAMPartialPolicy/platform-and-component-  Successful    InProgress              Ready                                     56m     reference LoggingLogSink logging/platfor
projects    IAMPartialPolicy/security-log-bucket-wri  Successful    InProgress              Ready                                     56m     reference LoggingLogSink logging/org-log
projects    IAMPolicyMember/logging-sa-monitoring-ad  Successful    Current                 Ready                                     62m     Resource is Current                     
projects    IAMPolicyMember/logging-sa-storageadmin-  Successful    Current                 Ready                                     62m     Resource is Current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    62m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    62m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    62m     Resource is current                     
projects    Project/dns-project-cso1                  Successful    Current                 Ready                                     56m     Resource is Current                     
projects    Project/logging-project-cso1              Successful    Current                 Ready                                     62m     Resource is Current                     
projects    Service/dns-project-cso1-dns              Pending       Unknown                 -                                         -                                               
projects    Service/logging-project-cso1-logging      Successful    Current                 Ready                                     56m     Resource is Current                     
projects    Service/logging-project-cso1-monitoring   Successful    Current                 Ready                                     56m     Resource is Current        

checking

LoggingLogSink/logging-project-cso1-data-access-sink
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl describe logginglogsink.logging.cnrm.cloud.google.com/logging-project-cso1-data-access-sink -n logging
Error from server (NotFound): logginglogsinks.logging.cnrm.cloud.google.com "logging-project-cso1-data-access-sink" not found

check the cluster

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true
NAME                                                                               CREATED AT
accesscontextmanageraccesslevels.accesscontextmanager.cnrm.cloud.google.com        2024-01-27T13:40:59Z
accesscontextmanageraccesspolicies.accesscontextmanager.cnrm.cloud.google.com      2024-01-27T13:40:59Z
accesscontextmanagerserviceperimeters.accesscontextmanager.cnrm.cloud.google.com   2024-01-27T13:40:59Z
alloydbbackups.alloydb.cnrm.cloud.google.com                                       2024-01-27T13:40:59Z
alloydbclusters.alloydb.cnrm.cloud.google.com                                      2024-01-27T13:41:00Z
alloydbinstances.alloydb.cnrm.cloud.google.com                                     2024-01-27T13:41:00Z
alloydbusers.alloydb.cnrm.cloud.google.com                                         2024-01-27T13:41:00Z
apigeeenvironments.apigee.cnrm.cloud.google.com                                    2024-01-27T13:41:00Z
apigeeorganizations.apigee.cnrm.cloud.google.com                                   2024-01-27T13:41:00Z
artifactregistryrepositories.artifactregistry.cnrm.cloud.google.com                2024-01-27T13:41:00Z
bigquerydatasets.bigquery.cnrm.cloud.google.com                                    2024-01-27T13:41:01Z
bigqueryjobs.bigquery.cnrm.cloud.google.com                                        2024-01-27T13:41:01Z
bigqueryroutines.bigquery.cnrm.cloud.google.com                                    2024-01-27T13:41:01Z
bigquerytables.bigquery.cnrm.cloud.google.com                                      2024-01-27T13:41:01Z
bigtableappprofiles.bigtable.cnrm.cloud.google.com                                 2024-01-27T13:41:01Z
bigtablegcpolicies.bigtable.cnrm.cloud.google.com                                  2024-01-27T13:41:01Z
bigtableinstances.bigtable.cnrm.cloud.google.com                                   2024-01-27T13:41:01Z
bigtabletables.bigtable.cnrm.cloud.google.com                                      2024-01-27T13:41:01Z
billingbudgetsbudgets.billingbudgets.cnrm.cloud.google.com                         2024-01-27T13:41:01Z
binaryauthorizationattestors.binaryauthorization.cnrm.cloud.google.com             2024-01-27T13:41:02Z
binaryauthorizationpolicies.binaryauthorization.cnrm.cloud.google.com              2024-01-27T13:41:02Z
certificatemanagercertificatemapentries.certificatemanager.cnrm.cloud.google.com   2024-01-27T13:41:02Z
certificatemanagercertificatemaps.certificatemanager.cnrm.cloud.google.com         2024-01-27T13:41:02Z
certificatemanagercertificates.certificatemanager.cnrm.cloud.google.com            2024-01-27T13:41:03Z
certificatemanagerdnsauthorizations.certificatemanager.cnrm.cloud.google.com       2024-01-27T13:41:03Z
cloudbuildtriggers.cloudbuild.cnrm.cloud.google.com                                2024-01-27T13:41:03Z
cloudfunctionsfunctions.cloudfunctions.cnrm.cloud.google.com                       2024-01-27T13:41:03Z
cloudidentitygroups.cloudidentity.cnrm.cloud.google.com                            2024-01-27T13:41:03Z
cloudidentitymemberships.cloudidentity.cnrm.cloud.google.com                       2024-01-27T13:41:04Z
cloudschedulerjobs.cloudscheduler.cnrm.cloud.google.com                            2024-01-27T13:41:04Z
computeaddresses.compute.cnrm.cloud.google.com                                     2024-01-27T13:41:04Z
computebackendbuckets.compute.cnrm.cloud.google.com                                2024-01-27T13:41:05Z
computebackendservices.compute.cnrm.cloud.google.com                               2024-01-27T13:41:05Z
computedisks.compute.cnrm.cloud.google.com                                         2024-01-27T13:41:05Z
computeexternalvpngateways.compute.cnrm.cloud.google.com                           2024-01-27T13:41:05Z
computefirewallpolicies.compute.cnrm.cloud.google.com                              2024-01-27T13:41:05Z
computefirewallpolicyassociations.compute.cnrm.cloud.google.com                    2024-01-27T13:41:05Z
computefirewallpolicyrules.compute.cnrm.cloud.google.com                           2024-01-27T13:41:05Z
computefirewalls.compute.cnrm.cloud.google.com                                     2024-01-27T13:41:06Z
computeforwardingrules.compute.cnrm.cloud.google.com                               2024-01-27T13:41:06Z
computehealthchecks.compute.cnrm.cloud.google.com                                  2024-01-27T13:41:06Z
computehttphealthchecks.compute.cnrm.cloud.google.com                              2024-01-27T13:41:06Z
computehttpshealthchecks.compute.cnrm.cloud.google.com                             2024-01-27T13:41:06Z
computeimages.compute.cnrm.cloud.google.com                                        2024-01-27T13:41:06Z
computeinstancegroupmanagers.compute.cnrm.cloud.google.com                         2024-01-27T13:41:06Z
computeinstancegroups.compute.cnrm.cloud.google.com                                2024-01-27T13:41:06Z
computeinstances.compute.cnrm.cloud.google.com                                     2024-01-27T13:41:06Z
computeinstancetemplates.compute.cnrm.cloud.google.com                             2024-01-27T13:41:06Z
computeinterconnectattachments.compute.cnrm.cloud.google.com                       2024-01-27T13:41:07Z
computenetworkendpointgroups.compute.cnrm.cloud.google.com                         2024-01-27T13:41:07Z
computenetworkpeerings.compute.cnrm.cloud.google.com                               2024-01-27T13:41:07Z
computenetworks.compute.cnrm.cloud.google.com                                      2024-01-27T13:41:08Z
computenodegroups.compute.cnrm.cloud.google.com                                    2024-01-27T13:41:08Z
computenodetemplates.compute.cnrm.cloud.google.com                                 2024-01-27T13:41:08Z
computepacketmirrorings.compute.cnrm.cloud.google.com                              2024-01-27T13:41:08Z
computeprojectmetadatas.compute.cnrm.cloud.google.com                              2024-01-27T13:41:08Z
computeregionnetworkendpointgroups.compute.cnrm.cloud.google.com                   2024-01-27T13:41:08Z
computereservations.compute.cnrm.cloud.google.com                                  2024-01-27T13:41:08Z
computeresourcepolicies.compute.cnrm.cloud.google.com                              2024-01-27T13:41:08Z
computerouterinterfaces.compute.cnrm.cloud.google.com                              2024-01-27T13:41:08Z
computerouternats.compute.cnrm.cloud.google.com                                    2024-01-27T13:41:08Z
computerouterpeers.compute.cnrm.cloud.google.com                                   2024-01-27T13:41:09Z
computerouters.compute.cnrm.cloud.google.com                                       2024-01-27T13:41:09Z
computeroutes.compute.cnrm.cloud.google.com                                        2024-01-27T13:41:09Z
computesecuritypolicies.compute.cnrm.cloud.google.com                              2024-01-27T13:41:09Z
computeserviceattachments.compute.cnrm.cloud.google.com                            2024-01-27T13:41:09Z
computesharedvpchostprojects.compute.cnrm.cloud.google.com                         2024-01-27T13:41:09Z
computesharedvpcserviceprojects.compute.cnrm.cloud.google.com                      2024-01-27T13:41:10Z
computesnapshots.compute.cnrm.cloud.google.com                                     2024-01-27T13:41:10Z
computesslcertificates.compute.cnrm.cloud.google.com                               2024-01-27T13:41:10Z
computesslpolicies.compute.cnrm.cloud.google.com                                   2024-01-27T13:41:10Z
computesubnetworks.compute.cnrm.cloud.google.com                                   2024-01-27T13:41:10Z
computetargetgrpcproxies.compute.cnrm.cloud.google.com                             2024-01-27T13:41:10Z
computetargethttpproxies.compute.cnrm.cloud.google.com                             2024-01-27T13:41:11Z
computetargethttpsproxies.compute.cnrm.cloud.google.com                            2024-01-27T13:41:11Z
computetargetinstances.compute.cnrm.cloud.google.com                               2024-01-27T13:41:11Z
computetargetpools.compute.cnrm.cloud.google.com                                   2024-01-27T13:41:11Z
computetargetsslproxies.compute.cnrm.cloud.google.com                              2024-01-27T13:41:11Z
computetargettcpproxies.compute.cnrm.cloud.google.com                              2024-01-27T13:41:11Z
computetargetvpngateways.compute.cnrm.cloud.google.com                             2024-01-27T13:41:11Z
computeurlmaps.compute.cnrm.cloud.google.com                                       2024-01-27T13:41:11Z
computevpngateways.compute.cnrm.cloud.google.com                                   2024-01-27T13:41:12Z
computevpntunnels.compute.cnrm.cloud.google.com                                    2024-01-27T13:41:12Z
configcontrollerinstances.configcontroller.cnrm.cloud.google.com                   2024-01-27T13:41:13Z
containeranalysisnotes.containeranalysis.cnrm.cloud.google.com                     2024-01-27T13:41:13Z
containerattachedclusters.containerattached.cnrm.cloud.google.com                  2024-01-27T13:41:13Z
containerclusters.container.cnrm.cloud.google.com                                  2024-01-27T13:41:13Z
containernodepools.container.cnrm.cloud.google.com                                 2024-01-27T13:41:13Z
datacatalogpolicytags.datacatalog.cnrm.cloud.google.com                            2024-01-27T13:41:14Z
datacatalogtaxonomies.datacatalog.cnrm.cloud.google.com                            2024-01-27T13:41:14Z
dataflowflextemplatejobs.dataflow.cnrm.cloud.google.com                            2024-01-27T13:41:14Z
dataflowjobs.dataflow.cnrm.cloud.google.com                                        2024-01-27T13:41:14Z
datafusioninstances.datafusion.cnrm.cloud.google.com                               2024-01-27T13:41:14Z
dataprocautoscalingpolicies.dataproc.cnrm.cloud.google.com                         2024-01-27T13:41:14Z
dataprocclusters.dataproc.cnrm.cloud.google.com                                    2024-01-27T13:41:15Z
dataprocworkflowtemplates.dataproc.cnrm.cloud.google.com                           2024-01-27T13:41:15Z
dlpdeidentifytemplates.dlp.cnrm.cloud.google.com                                   2024-01-27T13:41:16Z
dlpinspecttemplates.dlp.cnrm.cloud.google.com                                      2024-01-27T13:41:16Z
dlpjobtriggers.dlp.cnrm.cloud.google.com                                           2024-01-27T13:41:16Z
dlpstoredinfotypes.dlp.cnrm.cloud.google.com                                       2024-01-27T13:41:17Z
dnsmanagedzones.dns.cnrm.cloud.google.com                                          2024-01-27T13:41:17Z
dnspolicies.dns.cnrm.cloud.google.com                                              2024-01-27T13:41:17Z
dnsrecordsets.dns.cnrm.cloud.google.com                                            2024-01-27T13:41:17Z
edgecontainerclusters.edgecontainer.cnrm.cloud.google.com                          2024-01-27T13:41:17Z
edgecontainernodepools.edgecontainer.cnrm.cloud.google.com                         2024-01-27T13:41:17Z
edgecontainervpnconnections.edgecontainer.cnrm.cloud.google.com                    2024-01-27T13:41:17Z
edgenetworknetworks.edgenetwork.cnrm.cloud.google.com                              2024-01-27T13:41:17Z
edgenetworksubnets.edgenetwork.cnrm.cloud.google.com                               2024-01-27T13:41:18Z
eventarctriggers.eventarc.cnrm.cloud.google.com                                    2024-01-27T13:41:18Z
filestorebackups.filestore.cnrm.cloud.google.com                                   2024-01-27T13:41:18Z
filestoreinstances.filestore.cnrm.cloud.google.com                                 2024-01-27T13:41:18Z
firestoreindexes.firestore.cnrm.cloud.google.com                                   2024-01-27T13:41:19Z
folders.resourcemanager.cnrm.cloud.google.com                                      2024-01-27T13:41:19Z
gkehubfeaturememberships.gkehub.cnrm.cloud.google.com                              2024-01-27T13:41:19Z
gkehubfeatures.gkehub.cnrm.cloud.google.com                                        2024-01-27T13:41:19Z
gkehubmemberships.gkehub.cnrm.cloud.google.com                                     2024-01-27T13:41:19Z
iamaccessboundarypolicies.iam.cnrm.cloud.google.com                                2024-01-27T13:41:19Z
iamauditconfigs.iam.cnrm.cloud.google.com                                          2024-01-27T13:41:20Z
iamcustomroles.iam.cnrm.cloud.google.com                                           2024-01-27T13:41:20Z
iampartialpolicies.iam.cnrm.cloud.google.com                                       2024-01-27T13:41:20Z
iampolicies.iam.cnrm.cloud.google.com                                              2024-01-27T13:41:20Z
iampolicymembers.iam.cnrm.cloud.google.com                                         2024-01-27T13:41:20Z
iamserviceaccountkeys.iam.cnrm.cloud.google.com                                    2024-01-27T13:41:20Z
iamserviceaccounts.iam.cnrm.cloud.google.com                                       2024-01-27T13:41:20Z
iamworkforcepoolproviders.iam.cnrm.cloud.google.com                                2024-01-27T13:41:20Z
iamworkforcepools.iam.cnrm.cloud.google.com                                        2024-01-27T13:41:20Z
iamworkloadidentitypoolproviders.iam.cnrm.cloud.google.com                         2024-01-27T13:41:20Z
iamworkloadidentitypools.iam.cnrm.cloud.google.com                                 2024-01-27T13:41:20Z
iapbrands.iap.cnrm.cloud.google.com                                                2024-01-27T13:41:20Z
iapidentityawareproxyclients.iap.cnrm.cloud.google.com                             2024-01-27T13:41:20Z
identityplatformconfigs.identityplatform.cnrm.cloud.google.com                     2024-01-27T13:41:21Z
identityplatformoauthidpconfigs.identityplatform.cnrm.cloud.google.com             2024-01-27T13:41:21Z
identityplatformtenantoauthidpconfigs.identityplatform.cnrm.cloud.google.com       2024-01-27T13:41:21Z
identityplatformtenants.identityplatform.cnrm.cloud.google.com                     2024-01-27T13:41:21Z
kmscryptokeys.kms.cnrm.cloud.google.com                                            2024-01-27T13:41:21Z
kmskeyrings.kms.cnrm.cloud.google.com                                              2024-01-27T13:41:21Z
logginglogbuckets.logging.cnrm.cloud.google.com                                    2024-01-27T13:41:21Z
logginglogexclusions.logging.cnrm.cloud.google.com                                 2024-01-27T13:41:21Z
logginglogmetrics.logging.cnrm.cloud.google.com                                    2024-01-27T13:41:21Z
logginglogsinks.logging.cnrm.cloud.google.com                                      2024-01-27T13:41:21Z
logginglogviews.logging.cnrm.cloud.google.com                                      2024-01-27T13:41:21Z
memcacheinstances.memcache.cnrm.cloud.google.com                                   2024-01-27T13:41:22Z
monitoringalertpolicies.monitoring.cnrm.cloud.google.com                           2024-01-27T13:41:22Z
monitoringdashboards.monitoring.cnrm.cloud.google.com                              2024-01-27T13:41:22Z
monitoringgroups.monitoring.cnrm.cloud.google.com                                  2024-01-27T13:41:23Z
monitoringmetricdescriptors.monitoring.cnrm.cloud.google.com                       2024-01-27T13:41:23Z
monitoringmonitoredprojects.monitoring.cnrm.cloud.google.com                       2024-01-27T13:41:23Z
monitoringnotificationchannels.monitoring.cnrm.cloud.google.com                    2024-01-27T13:41:23Z
monitoringservicelevelobjectives.monitoring.cnrm.cloud.google.com                  2024-01-27T13:41:24Z
monitoringservices.monitoring.cnrm.cloud.google.com                                2024-01-27T13:41:24Z
monitoringuptimecheckconfigs.monitoring.cnrm.cloud.google.com                      2024-01-27T13:41:24Z
networkconnectivityhubs.networkconnectivity.cnrm.cloud.google.com                  2024-01-27T13:41:24Z
networkconnectivityspokes.networkconnectivity.cnrm.cloud.google.com                2024-01-27T13:41:24Z
networksecurityauthorizationpolicies.networksecurity.cnrm.cloud.google.com         2024-01-27T13:41:24Z
networksecurityclienttlspolicies.networksecurity.cnrm.cloud.google.com             2024-01-27T13:41:24Z
networksecurityservertlspolicies.networksecurity.cnrm.cloud.google.com             2024-01-27T13:41:24Z
networkservicesendpointpolicies.networkservices.cnrm.cloud.google.com              2024-01-27T13:41:24Z
networkservicesgateways.networkservices.cnrm.cloud.google.com                      2024-01-27T13:41:25Z
networkservicesgrpcroutes.networkservices.cnrm.cloud.google.com                    2024-01-27T13:41:25Z
networkserviceshttproutes.networkservices.cnrm.cloud.google.com                    2024-01-27T13:41:25Z
networkservicesmeshes.networkservices.cnrm.cloud.google.com                        2024-01-27T13:41:26Z
networkservicestcproutes.networkservices.cnrm.cloud.google.com                     2024-01-27T13:41:26Z
networkservicestlsroutes.networkservices.cnrm.cloud.google.com                     2024-01-27T13:41:26Z
osconfigguestpolicies.osconfig.cnrm.cloud.google.com                               2024-01-27T13:41:26Z
osconfigospolicyassignments.osconfig.cnrm.cloud.google.com                         2024-01-27T13:41:26Z
privatecacapools.privateca.cnrm.cloud.google.com                                   2024-01-27T13:41:27Z
privatecacertificateauthorities.privateca.cnrm.cloud.google.com                    2024-01-27T13:41:27Z
privatecacertificates.privateca.cnrm.cloud.google.com                              2024-01-27T13:41:27Z
privatecacertificatetemplates.privateca.cnrm.cloud.google.com                      2024-01-27T13:41:27Z
projects.resourcemanager.cnrm.cloud.google.com                                     2024-01-27T13:41:27Z
pubsublitereservations.pubsublite.cnrm.cloud.google.com                            2024-01-27T13:41:28Z
pubsubschemas.pubsub.cnrm.cloud.google.com                                         2024-01-27T13:41:28Z
pubsubsubscriptions.pubsub.cnrm.cloud.google.com                                   2024-01-27T13:41:28Z
pubsubtopics.pubsub.cnrm.cloud.google.com                                          2024-01-27T13:41:28Z
recaptchaenterprisekeys.recaptchaenterprise.cnrm.cloud.google.com                  2024-01-27T13:41:28Z
redisinstances.redis.cnrm.cloud.google.com                                         2024-01-27T13:41:28Z
resourcemanagerliens.resourcemanager.cnrm.cloud.google.com                         2024-01-27T13:41:28Z
resourcemanagerpolicies.resourcemanager.cnrm.cloud.google.com                      2024-01-27T13:41:29Z
runjobs.run.cnrm.cloud.google.com                                                  2024-01-27T13:41:29Z
runservices.run.cnrm.cloud.google.com                                              2024-01-27T13:41:29Z
secretmanagersecrets.secretmanager.cnrm.cloud.google.com                           2024-01-27T13:41:29Z
secretmanagersecretversions.secretmanager.cnrm.cloud.google.com                    2024-01-27T13:41:29Z
servicedirectoryendpoints.servicedirectory.cnrm.cloud.google.com                   2024-01-27T13:41:30Z
servicedirectorynamespaces.servicedirectory.cnrm.cloud.google.com                  2024-01-27T13:41:30Z
servicedirectoryservices.servicedirectory.cnrm.cloud.google.com                    2024-01-27T13:41:30Z
serviceidentities.serviceusage.cnrm.cloud.google.com                               2024-01-27T13:41:30Z
servicenetworkingconnections.servicenetworking.cnrm.cloud.google.com               2024-01-27T13:41:30Z
services.serviceusage.cnrm.cloud.google.com                                        2024-01-27T13:41:30Z
sourcereporepositories.sourcerepo.cnrm.cloud.google.com                            2024-01-27T13:41:30Z
spannerdatabases.spanner.cnrm.cloud.google.com                                     2024-01-27T13:41:30Z
spannerinstances.spanner.cnrm.cloud.google.com                                     2024-01-27T13:41:30Z
sqldatabases.sql.cnrm.cloud.google.com                                             2024-01-27T13:41:30Z
sqlinstances.sql.cnrm.cloud.google.com                                             2024-01-27T13:41:31Z
sqlsslcerts.sql.cnrm.cloud.google.com                                              2024-01-27T13:41:31Z
sqlusers.sql.cnrm.cloud.google.com                                                 2024-01-27T13:41:31Z
storagebucketaccesscontrols.storage.cnrm.cloud.google.com                          2024-01-27T13:41:31Z
storagebuckets.storage.cnrm.cloud.google.com                                       2024-01-27T13:41:31Z
storagedefaultobjectaccesscontrols.storage.cnrm.cloud.google.com                   2024-01-27T13:41:31Z
storagenotifications.storage.cnrm.cloud.google.com                                 2024-01-27T13:41:31Z
storagetransferjobs.storagetransfer.cnrm.cloud.google.com                          2024-01-27T13:41:31Z
tagstagbindings.tags.cnrm.cloud.google.com                                         2024-01-27T13:41:31Z
tagstagkeys.tags.cnrm.cloud.google.com                                             2024-01-27T13:41:31Z
tagstagvalues.tags.cnrm.cloud.google.com                                           2024-01-27T13:41:32Z
vpcaccessconnectors.vpcaccess.cnrm.cloud.google.com                                2024-01-27T13:41:32Z

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#continue-kpt-fn-render-after-failed-services-fixed

reference https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt#applying_a_kpt_package https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources

[obriensystems]

review 0.7.0 issue in dec https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/752

[obriensystems]

• needed org/org-policies as well as a min 1hour wait state

  logginglogsink.logging.cnrm.cloud.google.com/logging-project-oi0130-data-access-sink                16h   True    UpToDate   16h
  logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-sink   16h   True    UpToDate   16h
  logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-data-access-logging-project-oi0130        16h   True    UpToDate   16h
  logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-security-logging-project-oi0130           16h   True    UpToDate   16h
  logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-infra-log-sink         16h   True    UpToDate   16h
  logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-log-sink               16h   True    UpToDate   16h
  dnsmanagedzone.dns.cnrm.cloud.google.com/dns-project-oi0130-standard-core-public-dns   16h   True    UpToDate   16h

recheck cluster - time heals - just needed an extra hour

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone 
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-36746767/logginglogbucket.logging.cnrm.cloud.google.com/logging/security-log-bucket is Current: Resource is Current
inventory-36746767/logginglogbucket.logging.cnrm.cloud.google.com/logging/platform-and-component-log-bucket-oi0130 is Current: Resource is Current
inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden
inventory-36746767/monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/logging/kcc-oi-7970 is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iamauditconfig.iam.cnrm.cloud.google.com/projects/logging-project-data-access-log-config is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-oi0130-data-access-sink is Current: Resource is Current
inventory-36746767/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-oi0130 is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-oi0130-logging is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-oi0130-monitoring is Current: Resource is Current
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is Current: Resource is Current
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-36746767/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-oi0130-standard-core-public-dns is Current: Resource is Current
inventory-36746767/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-oi0130 is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-oi0130-dns is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is Current: Resource is Current
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging-except-kcc-oi-7970 is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-kcc-oi-7970 is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-kcc-oi-7970 is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-cloudbilling is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-cloudresourcemanager is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-serviceusage is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-accesscontextmanager is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-anthos is Current: Resource is Current
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-metric-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/config-management-monitoring/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//hierarchy is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-monitoring-admin-kcc-oi-7970-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-monitoring-admin-logging-project-oi0130-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-storageadmin-logging-project-oi0130-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//logging is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//networking is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//policies is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//projects is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-subnetwork-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier4-secretmanager-admin is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpn-peer-ips is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-automatic-iam-grants-for-default-service-accounts is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-audit-logging-exemption is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-upload is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-oi0130 is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-oi0130 is Current: Resource is Current

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone | grep not
inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get namespaces
NAME                              STATUS   AGE
cnrm-system                       Active   17h
config-control                    Active   17h
config-management-monitoring      Active   17h
config-management-system          Active   17h
configconnector-operator-system   Active   17h
configsync-healthcheck-system     Active   17h
default                           Active   18h
gatekeeper-system                 Active   17h
gke-gmp-system                    Active   17h
gke-managed-filestorecsi          Active   17h
gmp-public                        Active   17h
hierarchy                         Active   17h
krmapihosting-monitoring          Active   17h
krmapihosting-system              Active   17h
kube-node-lease                   Active   18h
kube-public                       Active   18h
kube-system                       Active   18h
logging                           Active   17h
networking                        Active   17h
policies                          Active   17h
projects                          Active   17h
resource-group-system             Active   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   17h   True    UpToDate   17h

NAME                                                                                                                   AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   17h   True    UpToDate   16h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         17h   True    UpToDate   16h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               17h   True    UpToDate   16h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      17h   True    UpToDate   16h

NAME                                                                                                       AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi0130-permissions   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi0130-permissions       17h   True    UpToDate   17h

NAME                                                                   AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi0130       17h   True    UpToDate   17h
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi0130   17h   True    UpToDate   17h

NAME                                                                           AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/dns-project-oi0130-dns              16h   True    UpToDate   16h
service.serviceusage.cnrm.cloud.google.com/logging-project-oi0130-logging      17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/logging-project-oi0130-monitoring   17h   True    UpToDate   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n networking
NAME                                                                                   AGE   READY   STATUS     STATUS AGE
dnsmanagedzone.dns.cnrm.cloud.google.com/dns-project-oi0130-standard-core-public-dns   16h   True    UpToDate   16h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n logging
NAME                                                                                      AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi0130   17h   True    UpToDate   17h
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                        17h   True    UpToDate   17h

NAME                                                                                                AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/logging-project-oi0130-data-access-sink                16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-sink   16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-data-access-logging-project-oi0130        16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-security-logging-project-oi0130           16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-infra-log-sink         16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-log-sink               16h   True    UpToDate   16h

NAME                                                                      AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-7970   17h   True    UpToDate   17h

NAME                                                                       AGE   READY   STATUS         STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket   17h   False   UpdateFailed   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    17h   True    UpToDate   17h
folder.resourcemanager.cnrm.cloud.google.com/clients                   17h   True    UpToDate   17h
folder.resourcemanager.cnrm.cloud.google.com/services                  17h   True    UpToDate   17h
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   17h   True    UpToDate   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n policies
NAME                                                                                                                 AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access                   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization                    17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access                       17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging                      17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-7970   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6                        17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login                                 17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm                              17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-7970           17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage                         17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-7970      17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types        17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal                 17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering                             17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpn-peer-ips                            17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation                    17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects                           17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward                                17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access                            17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains                17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations                          17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains                        17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-automatic-iam-grants-for-default-service-accounts    17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-audit-logging-exemption                      17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation                 17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-upload                   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip                                   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention                         17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access                      17h   True    UpToDate   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n config-control
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   17h   True    UpToDate   17h

NAME                                                                                              AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding   17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding          17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding                 17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding                   17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding                17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding                  17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding                  17h   True    UpToDate   17h

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions                   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-7970-permissions                    17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              17h   True    UpToDate   17h

NAME                                                                     AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa   17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa          17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa                 17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa                   17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa                17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa                  17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa                  17h   True    UpToDate   17h

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-accesscontextmanager   17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-anthos                 17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-cloudbilling           17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-cloudresourcemanager   17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-serviceusage           17h   True    UpToDate   17h

looking into the single failure
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone | grep not
inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

rerunning on other cloud-setup
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@0.7.1
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.7.1 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ cp core-landing-zone/org/
custom-roles/  org-policies/  org-sink.yaml  
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ cp -R core-landing-zone/org/org-policies/ ../core-landing-zone/org/
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ cd ../
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render core-landing-zone --truncate-output=false

[obriensystems]

update failed to fix 12 on this particular cloud-setup org - like oi did - removing and redeploying core-landing-zone

• but the issue is likely missing IAM permissions on clean account cloud-setup.org - where an older org that even had an older hub-env is ok obrien.industries below

• michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ ./setup.sh -b kcc-cso -u cso -n false -c false -l false -h false -r true -d false -j false -p kcc-cso-4380
  deleting REL_SUB_PACKAGE: core-landing-zone
  delete phase started
  storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket delete successful
  service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-monitoring delete successful
  service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-logging delete successful
  project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1 delete successful
  logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket delete successful
  logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions delete successful
  iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config delete successful
  delete phase finished
  reconcile phase started
  storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket reconcile pending
  service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-monitoring reconcile pending
  service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-logging reconcile pending
  project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1 reconcile pending
  logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket reconcile pending
  logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions reconcile successful
  iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile pending
  storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket reconcile successful
  logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-cso1 reconcile successful
  logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket reconcile successful
  service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-monitoring reconcile successful
  service.serviceusage.cnrm.cloud.google.com/logging-project-cso1-logging reconcile successful
  project.resourcemanager.cnrm.cloud.google.com/dns-project-cso1 reconcile successful
  iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile successful
  reconcile phase finished
  delete phase started
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-upload delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-audit-logging-exemption delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-automatic-iam-grants-for-default-service-accounts delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpn-peer-ips delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380 delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380 delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization delete successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access delete successful
  project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1 delete successful
  folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure delete successful
  folder.resourcemanager.cnrm.cloud.google.com/services delete successful
  folder.resourcemanager.cnrm.cloud.google.com/clients delete successful
  folder.resourcemanager.cnrm.cloud.google.com/audits delete successful
  monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380 delete successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies delete successful
  rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking delete successful
  rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging delete successful
  rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects delete successful
  rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects delete successful
  rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies delete successful
  rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control delete successful
  rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging delete successful
  delete phase finished
  reconcile phase started
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-upload reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-audit-logging-exemption reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-automatic-iam-grants-for-default-service-accounts reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpn-peer-ips reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380 reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380 reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization reconcile pending
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access reconcile pending
  project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1 reconcile pending
  folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
  folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
  folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
  folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
  monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380 reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control reconcile successful
  rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile pending
  rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access reconcile successful
  folder.resourcemanager.cnrm.cloud.google.com/services reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward reconcile successful
  monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-cso-4380 reconcile successful
  folder.resourcemanager.cnrm.cloud.google.com/clients reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects reconcile successful
  folder.resourcemanager.cnrm.cloud.google.com/audits reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpn-peer-ips reconcile successful
  folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile successful
  project.resourcemanager.cnrm.cloud.google.com/logging-project-cso1 reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-cso-4380 reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-cso-4380 reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization reconcile successful
  resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-cso1-permissions reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-cso1-permissions reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  reconcile phase finished
  delete phase started
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-serviceusage delete successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudresourcemanager delete successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudbilling delete successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-anthos delete successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-accesscontextmanager delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa delete successful
  iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa delete successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions delete successful
  deleting REL_SUB_PACKAGE: core-landing-zoneiampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-cso-4380-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions delete successful
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding delete successful
  iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin delete successful
  iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com delete successful
  namespace/projects delete successful
  namespace/policies delete successful
  namespace/networking delete successful
  namespace/logging delete successful
  namespace/hierarchy delete successful
  delete phase finished
  reconcile phase started
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-serviceusage reconcile successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudresourcemanager reconcile successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-cloudbilling reconcile successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-anthos reconcile successful
  service.serviceusage.cnrm.cloud.google.com/kcc-cso-4380-accesscontextmanager reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile successful
  iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-cso-4380-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile pending
  iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile pending
  iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
  namespace/projects reconcile pending
  namespace/policies reconcile pending
  namespace/networking reconcile pending
  namespace/logging reconcile pending
  namespace/hierarchy reconcile pending
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile successful
  iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile successful
  iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile successful
  iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile successful
  namespace/logging reconcile successful
  namespace/networking reconcile successful
  namespace/hierarchy reconcile successful
  namespace/projects reconcile successful
  namespace/policies reconcile successful
  reconcile phase finished
  inventory update started
  inventory update finished
  delete result: 116 attempted, 116 successful, 0 skipped, 0 failed
  reconcile result: 116 attempted, 116 successful, 0 skipped, 0 failed, 0 timed out
  wait 60 sec for gcp services to finish deleting before an optional GKE cluster delete
  deleting REL_SUB_PACKAGE: core-landing-zonedeleting REL_SUB_PACKAGE: core-landing-zonedeleting REL_SUB_PACKAGE: core-landing-zone

clean up

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ kubectl get gcp
No resources found in config-control namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ kubectl get gcp -n projects
No resources found in projects namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ kubectl get gcp -n logging
No resources found in logging namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ kubectl get gcp -n heirarchy
No resources found in heirarchy namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ kubectl get gcp -n networking
No resources found in networking namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ kubectl get gcp -n policies
No resources found in policies namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ rm -rf ../../../
github/ kpt/    
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ rm -rf ../../../kpt/core-landing-zone/

increment prefix in vars.sh - for projects and buckets

export PREFIX=cso2

setup.sh code
  echo "kpt live init"
  kpt live init $REL_SUB_PACKAGE --namespace config-control
  # --force
  echo "kpt fn render"
  kpt fn render $REL_SUB_PACKAGE --truncate-output=false
  echo "kpt live apply after 60s wait"
  sleep 60
  kpt live apply $REL_SUB_PACKAGE --reconcile-timeout=15m --output=table
  echo "check status"
  kpt live status --inv-type remote --statuses InProgress,NotFound

rerun

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso)$ ./setup.sh -b kcc-cso -u cso2 -n false -c false -l true -h false -r false -d false -j false -p kcc-cso-4380

wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER: 343139601407
DIRECTORY_CUSTOMER_ID: C02w06bdi
generated derived setters-core-landing-zone.yaml
Directory kpt exists - using it
deploying core-landing-zone
get kpt release package solutions/core-landing-zone version 0.7.1
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.7.1 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
copy over generated setters.yaml
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 600ms
  Results:
    [info] spec.folderRef.external: set field value to "276061734969"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.projectRef.name: set field value to "logging-project-cso2"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.name: set field value to "platform-and-component-log-bucket-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.projectRef.name: set field value to "logging-project-cso2"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "logging-project-cso2"
    [info] metadata.name: set field value to "kcc-cso-4380"
    [info] spec.metricsScope: set field value to "location/global/metricsScopes/logging-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.resourceRef.name: set field value to "logging-project-cso2"
    [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "org-log-sink-security-logging-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.resourceRef.name: set field value to "logging-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.resourceRef.name: set field value to "logging-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.resourceRef.name: set field value to "logging-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.resourceRef.name: set field value to "logging-project-cso2"
    [info] metadata.name: set field value to "logging-project-cso2-data-access-sink"
    [info] spec.projectRef.name: set field value to "logging-project-cso2"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso2/locations/northamerica-northeast1/buckets/security-log-bucket"
    [info] metadata.name: set field value to "logging-project-cso2"
    [info] spec.name: set field value to "logging-project-cso2"
    [info] spec.billingAccountRef.external: set field value to "01B35D-D56E1A-BAE17A"
    [info] metadata.name: set field value to "logging-project-cso2-logging"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.projectRef.external: set field value to "logging-project-cso2"
    [info] metadata.name: set field value to "logging-project-cso2-monitoring"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2"
    [info] spec.projectRef.external: set field value to "logging-project-cso2"
    [info] spec.folderRef.external: set field value to "276061734969"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-cso2"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso2/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-cso2"
    [info] spec.folderRef.external: set field value to "276061734969"
    [info] metadata.name: set field value to "dns-project-cso2-standard-core-public-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-cso2"
    [info] spec.dnsName: set field value to "cloud-setup.org."
    [info] metadata.name: set field value to "dns-project-cso2"
    [info] spec.name: set field value to "dns-project-cso2"
    [info] spec.billingAccountRef.external: set field value to "01B35D-D56E1A-BAE17A"
    [info] metadata.name: set field value to "dns-project-cso2-dns"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-cso2"
    [info] spec.projectRef.external: set field value to "dns-project-cso2"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-cso2"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso2/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-cso2"
    [info] spec.folderRef.external: set field value to "276061734969"
    [info] metadata.name: set field value to "compute-disable-serial-port-logging-except-kcc-cso-4380"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.name: set field value to "compute-require-shielded-vm-except-kcc-cso-4380"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.name: set field value to "compute-restrict-cloud-nat-usage-except-kcc-cso-4380"
    [info] spec.listPolicy.allow.values[0]: set field value to "under:projects/kcc-cso-4380"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-cso2"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso2/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-cso2"
    [info] metadata.name: set field value to "kcc-cso-4380-cloudbilling"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.name: set field value to "kcc-cso-4380-cloudresourcemanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.name: set field value to "kcc-cso-4380-serviceusage"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.name: set field value to "kcc-cso-4380-accesscontextmanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.name: set field value to "kcc-cso-4380-anthos"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "kcc-cso-4380"
    [info] spec.member: set field value to "serviceAccount:config-mgmt-mon-default-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[config-management-monitoring/default]"
    [info] spec.googleServiceAccount: set field value to "config-mgmt-mon-default-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "kcc-cso-4380"
    [info] spec.member: set field value to "serviceAccount:gatekeeper-admin-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[gatekeeper-system/gatekeeper-admin]"
    [info] spec.googleServiceAccount: set field value to "gatekeeper-admin-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:hierarchy-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]"
    [info] spec.googleServiceAccount: set field value to "hierarchy-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "734065690346"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-monitoring-admin-kcc-cso-4380-permissions"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.resourceRef.external: set field value to "kcc-cso-4380"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-monitoring-admin-logging-project-cso2-permissions"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "logging-project-cso2"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-storageadmin-logging-project-cso2-permissions"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "logging-project-cso2"
    [info] spec.resourceRef.name: set field value to "logging-project-cso2"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]"
    [info] spec.googleServiceAccount: set field value to "logging-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "734065690346"
    [info] spec.member: set field value to "serviceAccount:service-343139601407@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "kcc-cso-4380"
    [info] spec.member: set field value to "serviceAccount:service-343139601407@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "kcc-cso-4380"
    [info] spec.member: set field value to "serviceAccount:service-343139601407@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "734065690346"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "734065690346"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]"
    [info] spec.googleServiceAccount: set field value to "networking-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "734065690346"
    [info] spec.member: set field value to "serviceAccount:policies-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]"
    [info] spec.googleServiceAccount: set field value to "policies-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "276061734969"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.resourceRef.external: set field value to "734065690346"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-cso-4380"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]"
    [info] spec.googleServiceAccount: set field value to "projects-sa@kcc-cso-4380.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.listPolicy.allow.values: set field value to "- \"under:organizations/734065690346\"\n"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/cos-cloud\"\n"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.listPolicy.allow.values: set field value to "- \"@cloud-setup.org\"\n"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.listPolicy.allow.values: set field value to "- \"C02w06bdi\"\n"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] metadata.name: set field value to "org-log-sink-security-logging-project-cso2"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso2/locations/northamerica-northeast1/buckets/security-log-bucket"
    [info] metadata.name: set field value to "org-log-sink-data-access-logging-project-cso2"
    [info] spec.organizationRef.external: set field value to "734065690346"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-cso2/locations/northamerica-northeast1/buckets/security-log-bucket"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply after 60s wait

apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
data:
  org-id: "734..46"
  lz-folder-id: "27..9"
  billing-id: "01B...7A"
  management-project-id: "kcc-cso-4380"
  management-project-number: "34...07"
  management-namespace: config-control
  allowed-trusted-image-projects: |
    - "projects/cos-cloud"
  allowed-contact-domains: |
    - "@cloud-setup.org"
  allowed-policy-domain-members: |
    - "C02w06bdi"
  allowed-vpc-peering: |
    - "under:organizations/73...6"
  logging-project-id: logging-project-cso2
  security-log-bucket: security-log-bucket-cso2
  platform-and-component-log-bucket: platform-and-component-log-bucket-cso2
  retention-locking-policy: "false"
  retention-in-days: "1"
  dns-project-id: dns-project-cso2
  dns-name: "cloud-setup.org."

[obriensystems]

1131 - 15 min apply started

[obriensystems]

1202 - better

hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    17m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    17m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    17m     Resource is current                     
hierarchy   Folder/audits                             Successful    Current                 Ready                                     17m     Resource is Current                     
hierarchy   Folder/clients                            Successful    Current                 Ready                                     17m     Resource is Current                     
hierarchy   Folder/services                           Successful    Current                 Ready                                     17m     Resource is Current                     
hierarchy   Folder/services-infrastructure            Successful    Current                 Ready                                     17m     Resource is Current                     
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    17m     status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Successful    Current                 Ready                                     15m     Resource is Current                     
logging     LoggingLogBucket/security-log-bucket      Successful    Current                 Ready                                     15m     Resource is Current                     
logging     LoggingLogSink/logging-project-cso2-data  Successful    Current                 Ready                                     27s     Resource is Current                     
logging     LoggingLogSink/mgmt-project-cluster-plat  Successful    Current                 Ready                                     27s     Resource is Current                     
logging     LoggingLogSink/org-log-sink-data-access-  Successful    Current                 Ready                                     27s     Resource is Current                     
logging     LoggingLogSink/org-log-sink-security-log  Successful    Current                 Ready                                     27s     Resource is Current                     
logging     LoggingLogSink/platform-and-component-se  Successful    Current                 Ready                                     26s     Resource is Current                     
logging     LoggingLogSink/platform-and-component-se  Successful    Current                 Ready                                     26s     Resource is Current                     
logging     MonitoringMonitoredProject/kcc-cso-4380   Successful    Current                 Ready                                     17m     Resource is Current                     
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    17m     Resource is current                     
logging     StorageBucket/security-incident-log-buck  Successful    Failed                  Ready                                     15m     Update call failed: error fetching live 
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    17m     status.healthy is true                  
networking  DNSManagedZone/dns-project-cso2-standard  Successful    Failed                  Ready                                     28s     Update call failed: error applying desir
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    17m     status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-os  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-c  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-c  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/iam-automatic-iam-  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-audit-  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/storage-public-acc  Successful    Current                 Ready                                     17m     Resource is Current                     
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Current                 Ready                                     17m     Resource is Current                     
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    17m     status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Successful    Current                 Ready                                     15m     Resource is Current                     
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Successful    Current                 Ready                                     15m     Resource is Current                     
projects    IAMPartialPolicy/platform-and-component-  Successful    InProgress              Ready                                     15m     reference LoggingLogSink logging/platfor
projects    IAMPartialPolicy/platform-and-component-  Successful    Current                 Ready                                     15m     Resource is Current                     
projects    IAMPartialPolicy/security-log-bucket-wri  Successful    Current                 Ready                                     15m     Resource is Current                     
projects    IAMPolicyMember/logging-sa-monitoring-ad  Successful    Current                 Ready                                     17m     Resource is Current                     
projects    IAMPolicyMember/logging-sa-storageadmin-  Successful    Current                 Ready                                     17m     Resource is Current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    17m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    17m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    17m     Resource is current                     
projects    Project/dns-project-cso2                  Successful    Current                 Ready                                     15m     Resource is Current                     
projects    Project/logging-project-cso2              Successful    Current                 Ready                                     17m     Resource is Current                     
projects    Service/dns-project-cso2-dns              Successful    Current                 Ready                                     26s     Resource is Current                     
projects    Service/logging-project-cso2-logging      Successful    Current                 Ready                                     15m     Resource is Current                     
projects    Service/logging-project-cso2-monitoring   Successful    Current                 Ready                                     15m     Resource is Current                     

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status core-landing-zone --inv-type remote --statuses InProgress,NotFound
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status core-landing-zone | grep error
inventory-36537147/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-cso-4380.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

Same issue on redeployed cloud-setup

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status core-landing-zone | grep error
inventory-36537147/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-cso-4380.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl describe storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket -n logging
Name:         security-incident-log-bucket
Namespace:    logging
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              cnrm.cloud.google.com/management-conflict-prevention-policy: none
              cnrm.cloud.google.com/project-id: logging-project-cso2
              cnrm.cloud.google.com/state-into-spec: merge
              config.k8s.io/owning-inventory: ec099affabc09ae4652ae62190d9b794c9ec63d1-1706718583884502216
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-cso2
              internal.kpt.dev/upstream-identifier: storage.cnrm.cloud.google.com|StorageBucket|logging|security-incident-log-bucket
API Version:  storage.cnrm.cloud.google.com/v1beta1
Kind:         StorageBucket
Metadata:
  Creation Timestamp:  2024-01-31T16:33:31Z
  Generation:          1
  Resource Version:    4501241
  UID:                 b6cc605b-ac0b-45ae-ab03-0854998ab193
Spec:
  Autoclass:
    Enabled:                 true
  Location:                  northamerica-northeast1
  Public Access Prevention:  enforced
  Retention Policy:
    Is Locked:                  false
    Retention Period:           86400
  Uniform Bucket Level Access:  true
Status:
  Conditions:
    Last Transition Time:  2024-01-31T16:33:31Z
    Message:               Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-cso-4380.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden
    Reason:                UpdateFailed
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason        Age                 From                      Message
  ----     ------        ----                ----                      -------
  Warning  UpdateFailed  93s (x22 over 33m)  storagebucket-controller  Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-cso-4380.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

  however the logging-sa is missing Storage Admin

logging-sa@kcc-cso-4380.iam.gserviceaccount.com | logging-sa | Logging AdminMonitoring Admin

https://cloud.google.com/storage/docs/access-control/iam-roles

Storage Admin (roles/storage.admin) | Grants full control of buckets, managed folders, and objects, including getting and setting object ACLs or IAM policies.When applied to an individual bucket, control applies only to the specified bucket and the managed folders and objects within the bucket. | firebase.projects.getorgpolicy.policy.get1resourcemanager.projects.get2resourcemanager.projects.list2storage.buckets.*storage.managedFolders.*storage.objects.*storage.multipartUploads.*
-- | -- | --

added to #801

[obriensystems]

Fix for core-landing-zone setters.yaml generation part of the script

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit (kcc-cso-4380)$ git diff
diff --git a/solutions/setup.sh b/solutions/setup.sh
index 68c2763..4509441 100755
--- a/solutions/setup.sh
+++ b/solutions/setup.sh
@@ -240,7 +240,6 @@ metadata: # kpt-merge: /setters
   name: setters
   annotations:
     config.kubernetes.io/local-config: "true"
-    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
 data: 
   org-id: "${ORG_ID}"
   lz-folder-id: "${ROOT_FOLDER_ID}"
@@ -257,10 +256,12 @@ data:
   allowed-vpc-peering: |
     - "under:organizations/${ORG_ID}"
   logging-project-id: logging-project-${PREFIX}
-  security-log-bucket: security-log-bucket-${PREFIX}
+  security-incident-log-bucket: security-incident-log-bucket-${PREFIX}
   platform-and-component-log-bucket: platform-and-component-log-bucket-${PREFIX}
   retention-locking-policy: "false"
   retention-in-days: "1"
+  security-incident-log-bucket-retention-locking-policy: "false"
+  security-incident-log-bucket-retention-in-seconds: "86400"  

rerun live apply

logging     StorageBucket/security-incident-log-buck  Successful    NotFound                -                                         -       Resource not found                      
logging     StorageBucket/security-incident-log-buck  Successful    Current                 Ready                                     3s      Resource is Current  

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n logging
NAME                                                                                      AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi0130   47h   True    UpToDate   47h
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                        47h   True    UpToDate   47h

NAME                                                                                                AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/logging-project-oi0130-data-access-sink                47h   True    UpToDate   47h
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-sink   47h   True    UpToDate   47h
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-data-access-logging-project-oi0130        47h   True    UpToDate   47h
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-security-logging-project-oi0130           47h   True    UpToDate   47h
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-infra-log-sink         47h   True    UpToDate   47h
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-log-sink               47h   True    UpToDate   47h

NAME                                                                      AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-7970   47h   True    UpToDate   47h

NAME                                                                              AGE     READY   STATUS     STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket-oi0130   3m44s   True    UpToDate   3m41s

[obriensystems]

testing client-setup script

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ ./setup.sh -b kcc-cso -u cso2 -n false -c false -l false -m true -o false -g false -h false -r false -d false -j false -p kcc-cso-4380

deploy client-setup
KCC_PROJECT_NUMBER: 343139601407
DIRECTORY_CUSTOMER_ID: C02w06bdi
generated derived setters-client-setup.yaml
Directory kpt exists - using it
deploying client-setup
get kpt release package solutions/client-setup version 0.7.1
Package "client-setup":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/client-setup/0.7.1 -> FETCH_HEAD
Adding package "solutions/client-setup".

Fetched 1 package(s).
copy over generated setters.yaml

apiVersion: v1
kind: ConfigMap
metadata: 
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data: 
  org-id: "734...6"
  management-project-id: "kcc-cso-4380"
  management-project-number: "34...07"
  management-namespace: config-control
  client-name: client-cso2
  client-billing-id: "01...17A"
  client-management-project-id: client-management-project-cso2
  repo-url: git-repo-to-observe
  repo-branch: main
  repo-dir: csync/deploy/env    
  dns-project-id: dns-project-cso2

kpt method not applicable
config-man  RootSync/client-cso2-csync                Successful    Failed                  Stalled                                   25s     Secret git-creds not found: create one t

raised #807

Missed some procedures in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-client.md

[obriensystems]

gitops approach for client-setup - manual first

remove lz

Remember to follow client-setup

Thanks Alain https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-client.md

[fmichaelobrien]

remove partial client-setup - rerun script with directory deletion first

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status client-setup
inventory-30609223/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients.client-cso2 is Current: Resource is Current
inventory-30609223/project.resourcemanager.cnrm.cloud.google.com/projects/client-management-project-cso2 is Current: Resource is Current
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-iam is NotFound: Resource not found
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-resourcemanager is NotFound: Resource not found
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-billing is NotFound: Resource not found
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-serviceusage is NotFound: Resource not found
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-container is NotFound: Resource not found
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-ids is NotFound: Resource not found
inventory-30609223/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso2-servicenetworking is NotFound: Resource not found
inventory-30609223/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-admin-sa is NotFound: Resource not found
inventory-30609223/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-admin-sa-workload-identity-binding is NotFound: Resource not found
inventory-30609223/namespace//client-cso2-admin is Current: Resource is current
inventory-30609223/configconnectorcontext.core.cnrm.cloud.google.com/client-cso2-admin/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-projects/allow-resource-reference-from-client-cso2-admin is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-admin/allow-resource-reference-from-client-cso2-projects is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-networking/allow-resource-reference-from-client-cso2-admin is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-admin/allow-resource-reference-from-client-cso2-networking is NotFound: Resource not found
inventory-30609223/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-hierarchy-sa is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-hierarchy-sa-folderadmin-permissions is NotFound: Resource not found
inventory-30609223/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-hierarchy-sa-workload-identity-binding is NotFound: Resource not found
inventory-30609223/namespace//client-cso2-hierarchy is Current: Resource is current
inventory-30609223/configconnectorcontext.core.cnrm.cloud.google.com/client-cso2-hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-hierarchy/allow-resource-reference-from-projects is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-resource-reference-from-client-cso2-hierarchy is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-hierarchy/allow-client-cso2-hierarchy-resource-reference-from-policies is NotFound: Resource not found
inventory-30609223/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-logging-sa is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-logging-sa-logadmin-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-logging-sa-monitoringadmin-permissions is NotFound: Resource not found
inventory-30609223/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-logging-sa-workload-identity-binding is NotFound: Resource not found
inventory-30609223/namespace//client-cso2-logging is Current: Resource is current
inventory-30609223/configconnectorcontext.core.cnrm.cloud.google.com/client-cso2-logging/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-projects/allow-resource-reference-from-logging is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/projects/config-control-sa-iamserviceaccountadmin-client-management-project-cso2-permissions is NotFound: Resource not found
inventory-30609223/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/client-cso2-config-control-sa is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/projects/client-cso2-config-control-sa-projectiamadmin-client-management-project-cso2-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-config-control-sa-iamserviceaccountadmin-client-folder-permissions is NotFound: Resource not found
inventory-30609223/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/client-cso2-config-control-sa-workload-identity-binding is NotFound: Resource not found
inventory-30609223/namespace//client-cso2-config-control is Current: Resource is current
inventory-30609223/configconnectorcontext.core.cnrm.cloud.google.com/client-cso2-config-control/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-30609223/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-networking-sa is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-networkadmin-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-security-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-dns-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/projects/client-cso2-networking-sa-tier2-dns-record-admin-permission is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-service-control-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/config-control/client-cso2-networking-sa-xpnadmin-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-servicedirectoryeditor-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-client-folder-org-resource-admin-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-networking-sa-cloudids-admin-permissions is NotFound: Resource not found
inventory-30609223/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-networking-sa-workload-identity-binding is NotFound: Resource not found
inventory-30609223/namespace//client-cso2-networking is Current: Resource is current
inventory-30609223/configconnectorcontext.core.cnrm.cloud.google.com/client-cso2-networking/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-projects/allow-resource-reference-from-networking is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/networking/allow-resource-reference-from-client-cso2-networking is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-hierarchy/allow-client-cso2-hierarchy-resource-reference-from-client-cso2-networking is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-client-cso2-networking is NotFound: Resource not found
inventory-30609223/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-projects-sa is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-projects-sa-projectiamadmin-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-projects-sa-projectcreator-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-projects-sa-projectmover-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-projects-sa-projectdeleter-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso2-projects-sa-serviceusageadmin-permissions is NotFound: Resource not found
inventory-30609223/iampolicymember.iam.cnrm.cloud.google.com/config-control/client-cso2-projects-sa-billinguser-permissions is NotFound: Resource not found
inventory-30609223/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso2-config-control/client-cso2-projects-sa-workload-identity-binding is NotFound: Resource not found
inventory-30609223/namespace//client-cso2-projects is Current: Resource is current
inventory-30609223/configconnectorcontext.core.cnrm.cloud.google.com/client-cso2-projects/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-30609223/rolebinding.rbac.authorization.k8s.io/client-cso2-config-control/allow-resource-reference-from-projects is NotFound: Resource not found
inventory-30609223/rootsync.configsync.gke.io/config-management-system/client-cso2-csync is Failed: Secret git-creds not found: create one to allow client authentication
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ 

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live destroy client-setup
delete result: 9 attempted, 9 successful, 0 skipped, 0 failed
reconcile result: 9 attempted, 9 successful, 0 skipped, 0 failed, 0 timed out

rerun with separate prefix for each package

export PREFIX=cso2
export PREFIX_CLIENT_SETUP=cso3
export PREFIX_CLIENT_LANDING_ZONE=cso3
export PREFIX_CLIENT_PROJECT_SETUP=cso3

  org-id: "${ORG_ID}"
  management-project-id: "${KCC_PROJECT_ID}"
  management-project-number: "${KCC_PROJECT_NUMBER}"
  management-namespace: config-control
  client-name: client-${PREFIX_CLIENT_SETUP}
  client-billing-id: "${BILLING_ID}"
  client-management-project-id: client-management-project-${PREFIX_CLIENT_SETUP}
  repo-url: git-repo-to-observe
  repo-branch: main
  repo-dir: csync/deploy/env    
  dns-project-id: dns-project-${PREFIX}

    echo "removing gitops directory"
  rm -rf $REL_SUB_PACKAGE/root-sync-git

rerunning script for client-setup

[fmichaelobrien]

client-setup status

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/client-cso3-admin               Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/client-cso3-config-control      Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/client-cso3-hierarchy           Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/client-cso3-logging             Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/client-cso3-networking          Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/client-cso3-projects            Successful    Current                 <None>                                    4m      Resource is current                     
client-cso  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
client-cso  IAMPartialPolicy/client-cso3-admin-sa-wo  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMPartialPolicy/client-cso3-hierarchy-s  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMPartialPolicy/client-cso3-logging-sa-  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMPartialPolicy/client-cso3-networking-  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMPartialPolicy/client-cso3-projects-sa  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMServiceAccount/client-cso3-admin-sa    Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMServiceAccount/client-cso3-hierarchy-  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMServiceAccount/client-cso3-logging-sa  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMServiceAccount/client-cso3-networking  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  IAMServiceAccount/client-cso3-projects-s  Successful    Current                 Ready                                     2m      Resource is Current                     
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
client-cso  RoleBinding/allow-client-cso3-hierarchy-  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  RoleBinding/allow-client-cso3-hierarchy-  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
client-cso  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    3m      Resource is current                     
client-cso  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    2m      Resource is current                     
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    2m      Resource is current                     
client-cso  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    2m      Resource is current                     
config-con  IAMPartialPolicy/client-cso3-config-cont  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/client-cso3-networking-s  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/client-cso3-projects-sa-  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/client-cso3-config-con  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-config-contr  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-hierarchy-sa  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-logging-sa-l  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-logging-sa-m  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Current                 Ready                                     2m      Resource is Current                     
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-networking-s  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-projects-sa-  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-projects-sa-  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-projects-sa-  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-projects-sa-  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   IAMPolicyMember/client-cso3-projects-sa-  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   Folder/clients.client-cso3                Successful    Current                 Ready                                     4m      Resource is Current                     
networking  RoleBinding/allow-resource-reference-fro  Successful    Current                 <None>                                    2m      Resource is current                     
projects    IAMPolicyMember/client-cso3-config-contr  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    IAMPolicyMember/client-cso3-networking-s  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    IAMPolicyMember/config-control-sa-iamser  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Project/client-management-project-cso3    Successful    Current                 Ready                                     4m      Resource is Current                     
projects    Service/client-management-project-cso3-b  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Service/client-management-project-cso3-c  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Service/client-management-project-cso3-i  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Service/client-management-project-cso3-i  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Service/client-management-project-cso3-r  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Service/client-management-project-cso3-s  Successful    Current                 Ready                                     2m      Resource is Current                     
projects    Service/client-management-project-cso3-s  Successful    Current                 Ready                                     2m      Resource is Current                     

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get namespaces | grep client
client-cso3-admin                 Active   6m23s
client-cso3-config-control        Active   6m23s
client-cso3-hierarchy             Active   6m23s
client-cso3-logging               Active   6m23s
client-cso3-networking            Active   6m23s
client-cso3-projects              Active   6m23s

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ 
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp | grep client
iampolicymember.iam.cnrm.cloud.google.com/client-cso3-networking-sa-xpnadmin-permissions                         4m9s   True    UpToDate   110s
iampolicymember.iam.cnrm.cloud.google.com/client-cso3-projects-sa-billinguser-permissions                        4m8s   True    UpToDate   105s
iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-config-control-sa   4m2s   True    UpToDate   4m
iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-config-control-sa-workload-identity-binding   4m9s   True    UpToDate   3m59s

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n client-cso3-config-control
NAME                                                                    AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-admin-sa        7m51s   True    UpToDate   6m42s
iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-hierarchy-sa    7m51s   True    UpToDate   6m42s
iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-logging-sa      7m51s   True    UpToDate   6m42s
iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-networking-sa   7m51s   True    UpToDate   6m42s
iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-projects-sa     7m50s   True    UpToDate   6m41s

NAME                                                                                             AGE     READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-admin-sa-workload-identity-binding        7m59s   True    UpToDate   6m41s
iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-hierarchy-sa-workload-identity-binding    7m59s   True    UpToDate   6m41s
iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-logging-sa-workload-identity-binding      7m59s   True    UpToDate   6m41s
iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-networking-sa-workload-identity-binding   7m58s   True    UpToDate   6m41s
iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-projects-sa-workload-identity-binding     7m58s   True    UpToDate   6m40s
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n client-cso3-admin
No resources found in client-cso3-admin namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n client-cso3-hierarchy
No resources found in client-cso3-hierarchy namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n client-cso3-logging
No resources found in client-cso3-logging namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n client-cso3-networking
No resources found in client-cso3-networking namespace.
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n client-cso3-projects
No resources found in client-cso3-projects namespace.

Some services are in the wrong namespace (core-landing-zone - not client-setup)

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n projects | grep client
iampolicymember.iam.cnrm.cloud.google.com/client-cso3-config-control-sa-projectiamadmin-client-management-project-cso3-permissions   10m   True    UpToDate   10m
iampolicymember.iam.cnrm.cloud.google.com/client-cso3-networking-sa-tier2-dns-record-admin-permission                                10m   True    UpToDate   9m19s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-iamserviceaccountadmin-client-management-project-cso3-permissions        10m   True    UpToDate   10m
project.resourcemanager.cnrm.cloud.google.com/client-management-project-cso3   12m   True    UpToDate   11m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-billing             10m     True    UpToDate   10m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-container           10m     True    UpToDate   10m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-iam                 10m     True    UpToDate   10m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-ids                 10m     True    UpToDate   10m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-resourcemanager     10m     True    UpToDate   10m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-servicenetworking   10m     True    UpToDate   10m
service.serviceusage.cnrm.cloud.google.com/client-management-project-cso3-serviceusage        10m     True    UpToDate   10m
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kubectl get gcp -n logging | grep client

[fmichaelobrien]

check kpt live status for client-setup - looks OK

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kpt live status client-setup --inv-type remote
inventory-36537147/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-36537147/logginglogbucket.logging.cnrm.cloud.google.com/logging/security-log-bucket is Current: Resource is Current
inventory-36537147/logginglogbucket.logging.cnrm.cloud.google.com/logging/platform-and-component-log-bucket-cso2 is Current: Resource is Current
inventory-36537147/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket-cso2 is Current: Resource is Current
inventory-36537147/monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/logging/kcc-cso-4380 is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is Current: Resource is Current
inventory-36537147/iamauditconfig.iam.cnrm.cloud.google.com/projects/logging-project-data-access-log-config is Current: Resource is Current
inventory-36537147/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-cso2-data-access-sink is Current: Resource is Current
inventory-36537147/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-cso2 is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-cso2-logging is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-cso2-monitoring is Current: Resource is Current
inventory-36537147/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-36537147/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is Current: Resource is Current
inventory-36537147/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-36537147/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-cso2-standard-core-public-dns is Current: Resource is Current
inventory-36537147/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-cso2 is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-cso2-dns is Current: Resource is Current
inventory-36537147/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is Current: Resource is Current
inventory-36537147/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging-except-kcc-cso-4380 is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-kcc-cso-4380 is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-kcc-cso-4380 is Current: Resource is Current
inventory-36537147/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-cloudbilling is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-cloudresourcemanager is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-serviceusage is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-accesscontextmanager is Current: Resource is Current
inventory-36537147/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-cso-4380-anthos is Current: Resource is Current
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-metric-writer-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/config-management-monitoring/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/namespace//hierarchy is Current: Resource is current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-36537147/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-36537147/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-36537147/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-monitoring-admin-kcc-cso-4380-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-monitoring-admin-logging-project-cso2-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-storageadmin-logging-project-cso2-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/namespace//logging is Current: Resource is current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/namespace//networking is Current: Resource is current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/namespace//policies is Current: Resource is current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-36537147/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-36537147/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-36537147/namespace//projects is Current: Resource is current
inventory-36537147/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36537147/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-36537147/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-36537147/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-subnetwork-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-36537147/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier4-secretmanager-admin is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpn-peer-ips is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-automatic-iam-grants-for-default-service-accounts is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-audit-logging-exemption is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-upload is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-36537147/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current
inventory-36537147/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-cso2 is Current: Resource is Current
inventory-36537147/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-cso2 is Current: Resource is Current
inventory-70330823/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients.client-cso3 is Current: Resource is Current
inventory-70330823/project.resourcemanager.cnrm.cloud.google.com/projects/client-management-project-cso3 is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-iam is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-resourcemanager is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-billing is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-serviceusage is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-container is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-ids is Current: Resource is Current
inventory-70330823/service.serviceusage.cnrm.cloud.google.com/projects/client-management-project-cso3-servicenetworking is Current: Resource is Current
inventory-70330823/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-admin-sa is Current: Resource is Current
inventory-70330823/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-70330823/namespace//client-cso3-admin is Current: Resource is current
inventory-70330823/configconnectorcontext.core.cnrm.cloud.google.com/client-cso3-admin/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-projects/allow-resource-reference-from-client-cso3-admin is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-admin/allow-resource-reference-from-client-cso3-projects is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-networking/allow-resource-reference-from-client-cso3-admin is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-admin/allow-resource-reference-from-client-cso3-networking is Current: Resource is current
inventory-70330823/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-hierarchy-sa is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-70330823/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-70330823/namespace//client-cso3-hierarchy is Current: Resource is current
inventory-70330823/configconnectorcontext.core.cnrm.cloud.google.com/client-cso3-hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-hierarchy/allow-resource-reference-from-projects is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-resource-reference-from-client-cso3-hierarchy is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-hierarchy/allow-client-cso3-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-70330823/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-logging-sa is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-logging-sa-logadmin-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-logging-sa-monitoringadmin-permissions is Current: Resource is Current
inventory-70330823/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-logging-sa-workload-identity-binding is Current: Resource is Current
inventory-70330823/namespace//client-cso3-logging is Current: Resource is current
inventory-70330823/configconnectorcontext.core.cnrm.cloud.google.com/client-cso3-logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-projects/allow-resource-reference-from-logging is Current: Resource is current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/projects/config-control-sa-iamserviceaccountadmin-client-management-project-cso3-permissions is Current: Resource is Current
inventory-70330823/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/client-cso3-config-control-sa is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/projects/client-cso3-config-control-sa-projectiamadmin-client-management-project-cso3-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-config-control-sa-iamserviceaccountadmin-client-folder-permissions is Current: Resource is Current
inventory-70330823/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/client-cso3-config-control-sa-workload-identity-binding is Current: Resource is Current
inventory-70330823/namespace//client-cso3-config-control is Current: Resource is current
inventory-70330823/configconnectorcontext.core.cnrm.cloud.google.com/client-cso3-config-control/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-70330823/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-networking-sa is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-security-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-dns-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/projects/client-cso3-networking-sa-tier2-dns-record-admin-permission is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-service-control-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/config-control/client-cso3-networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-client-folder-org-resource-admin-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-networking-sa-cloudids-admin-permissions is Current: Resource is Current
inventory-70330823/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-networking-sa-workload-identity-binding is Current: Resource is Current
inventory-70330823/namespace//client-cso3-networking is Current: Resource is current
inventory-70330823/configconnectorcontext.core.cnrm.cloud.google.com/client-cso3-networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-projects/allow-resource-reference-from-networking is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/networking/allow-resource-reference-from-client-cso3-networking is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-hierarchy/allow-client-cso3-hierarchy-resource-reference-from-client-cso3-networking is Current: Resource is current
inventory-70330823/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-client-cso3-networking is Current: Resource is current
inventory-70330823/iamserviceaccount.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-projects-sa is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-projects-sa-projectmover-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/hierarchy/client-cso3-projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-70330823/iampolicymember.iam.cnrm.cloud.google.com/config-control/client-cso3-projects-sa-billinguser-permissions is Current: Resource is Current
inventory-70330823/iampartialpolicy.iam.cnrm.cloud.google.com/client-cso3-config-control/client-cso3-projects-sa-workload-identity-binding is Current: Resource is Current
inventory-70330823/namespace//client-cso3-projects is Current: Resource is current
inventory-70330823/configconnectorcontext.core.cnrm.cloud.google.com/client-cso3-projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-70330823/rolebinding.rbac.authorization.k8s.io/client-cso3-config-control/allow-resource-reference-from-projects is Current: Resource is current

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ kpt live status client-setup --inv-type remote | grep not
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ 

[fmichaelobrien]

moving to client-landing-zone - then client-project-setup

client-landing-zone setters.yaml needs regionalization in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/791#issuecomment-1930391676

cat << EOF > ./${REL_SUB_PACKAGE}/setters-${REL_SUB_PACKAGE}.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  client-name: client-${PREFIX_CLIENT_LANDING_ZONE}
  client-billing-id: "${BILLING_ID}"
  client-folderviewer: 'user:${SUPER_ADMIN_EMAIL}'
  logging-project-id: logging-project-${PREFIX}
  retention-locking-policy: "false"
  retention-in-days: "1"
  host-project-id: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}
  project-allowed-restrict-vpc-peering: |
    - folders/${ROOT_FOLDER_ID}
#    - under:projects/PROJECT_ID
  standard-nonp-cidr: |
    - 10.1.0.0/18
    - 172.16.0.0/13
  standard-nane1-nonp-main-snet: 10.1.0.0/21
  standard-nane2-nonp-main-snet: 10.1.8.0/21
  standard-pbmm-cidr: |
    - 10.1.128.0/18
    - 172.24.0.0/13
  standard-nane1-pbmm-main-snet: 10.1.128.0/21
  standard-nane2-pbmm-main-snet: 10.1.136.0/21
  firewall-internal-ip-ranges: |
    - 10.0.0.0/8
    - 172.16.0.0/12
    - 192.168.0.0/16
  denied-sanctioned-countries: |
    - "XC"
  allowed-os-update-domains: |
    - "debian.map.fastlydns.net"
    - "debian.org"
    - "deb.debian.org"
    - "ubuntu.com"
    - "cloud.google.com"
    - "packages.cloud.google.com"
    - "security.ubuntu.com"
    - "northamerica-northeast1.gce.archive.ubuntu.com"
    - "northamerica-northeast2.gce.archive.ubuntu.com"
  allowed-os-update-source-ip-ranges: |
    - "10.1.0.0/21"
    - "10.1.8.0/21"
    - "10.1.32.0/19"
    - "10.1.128.0/21"
    - "10.1.136.0/21"
    - "10.1.160.0/19"
  dns-project-id: dns-project-${PREFIX}
  dns-name: "client-${PREFIX_CLIENT_LANDING_ZONE}.${CONTACT_DOMAIN}."
  dns-name: "${CONTACT_DOMAIN}."
  dns-nameservers: |
    - "ns-cloud-a1.googledomains.com."
    - "ns-cloud-a2.googledomains.com."
    - "ns-cloud-a3.googledomains.com."
    - "ns-cloud-a4.googledomains.com."
EOF

running

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ ./setup.sh -b kcc-cso -u cso2 -n false -c false -l false -m false -o true -g false -h false -r false -d false -j false -p kcc-cso-4380

apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  client-name: client-cso3
  client-billing-id: "01B..7A"
  client-folderviewer: 'user:mi..g'
  logging-project-id: logging-project-cso2
  retention-locking-policy: "false"
  retention-in-days: "1"
  host-project-id: net-host-project-cso3
  project-allowed-restrict-vpc-peering: |
    - folders/27...969
  #    - under:projects/PROJECT_ID
  standard-nonp-cidr: |
    - 10.1.0.0/18
    - 172.16.0.0/13
  standard-nane1-nonp-main-snet: 10.1.0.0/21
  standard-nane2-nonp-main-snet: 10.1.8.0/21
  standard-pbmm-cidr: |
    - 10.1.128.0/18
    - 172.24.0.0/13
  standard-nane1-pbmm-main-snet: 10.1.128.0/21
  standard-nane2-pbmm-main-snet: 10.1.136.0/21
  firewall-internal-ip-ranges: |
    - 10.0.0.0/8
    - 172.16.0.0/12
    - 192.168.0.0/16
  denied-sanctioned-countries: |
    - "XC"
  allowed-os-update-domains: |
    - "debian.map.fastlydns.net"
    - "debian.org"
    - "deb.debian.org"
    - "ubuntu.com"
    - "cloud.google.com"
    - "packages.cloud.google.com"
    - "security.ubuntu.com"
    - "northamerica-northeast1.gce.archive.ubuntu.com"
    - "northamerica-northeast2.gce.archive.ubuntu.com"
  allowed-os-update-source-ip-ranges: |
    - "10.1.0.0/21"
    - "10.1.8.0/21"
    - "10.1.32.0/19"
    - "10.1.128.0/21"
    - "10.1.136.0/21"
    - "10.1.160.0/19"
  dns-project-id: dns-project-cso2
  dns-name: "client-cso3.cloud-setup.org."
  dns-nameservers: |
    - "ns-cloud-a1.googledomains.com."
    - "ns-cloud-a2.googledomains.com."
    - "ns-cloud-a3.googledomains.com."
    - "ns-cloud-a4.googledomains.com."

[fmichaelobrien]

client-landing-zone gcloud workaround for PSC forwarding rule

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/823

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-client.md#add-the-client-landing-zone-package

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ HOST_PROJECT_ID='net-host-project-cso3'
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ gcloud compute forwarding-rules create standardpscapisfw \
  --global \
  --network=global-standard-vpc \
  --address=standard-psc-apis-ip \
  --target-google-apis-bundle=all-apis \
  --project=${HOST_PROJECT_ID} \
  --service-directory-registration=projects/${HOST_PROJECT_ID}/locations/northamerica-northeast1
Created [https://www.googleapis.com/compute/v1/projects/net-host-project-cso3/global/forwardingRules/standardpscapisfw].

verify https://cloud.google.com/sdk/gcloud/reference/compute/forwarding-rules/list

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ gcloud compute forwarding-rules list --project ${HOST_PROJECT_ID}
NAME: standardpscapisfw
REGION: 
IP_ADDRESS: 10.255.255.254
IP_PROTOCOL: TCP
TARGET: all-apis

Check config-controller acquire of resource

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status client-landing-zone | grep computeforwardingrule
inventory-90770020/computeforwardingrule.compute.cnrm.cloud.google.com/client-cso3-networking/net-host-project-cso3-standard-psc-apis-fw is Current: Resource is Current

[obriensystems]

client-project-setup

working script

spawning

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/830 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/829

from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26

  name: net-host-project-cso3-nane1-standard-nonp-main-snet # kpt-set: ${host-project-id}-nane1-standard-nonp-main-snet
${host-project-id}-nane1-standard-nonp-main-snet

testing values.yaml generation

cat << EOF > ./${REL_SUB_PACKAGE}/setters-${REL_SUB_PACKAGE}.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  org-id: "${ORG_ID}"  
  management-project-id: "${KCC_PROJECT_ID}"
  management-namespace: "${MANAGEMENT_NAMESPACE}"
  client-name: '${CLIENT_NAME_1}'
  client-management-project-id: client-management-project-${PREFIX_CLIENT_SETUP}
  host-project-id: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}
  # see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26
  allowed-nane1-main-subnet: ${host-project-id}-nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane2-main-subnet: ${host-project-id}-nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  project-id: client-project-${PREFIX_CLIENT_PROJECT_SETUP}
  project-billing-id: "${BILLING_ID}"
  project-parent-folder: standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}
  repo-url: git-repo-to-observe
  repo-branch: main
  tier3-repo-dir: csync/tier3/configcontroller/deploy/env
  tier4-repo-dir: csync/tier4/configcontroller/deploy/env
EOF

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ ./setup.sh -b kcc-cso -u cso2 -n false -c false -l false -m false -o false -s true -g false -h false -r false -d false -j false -p kcc-cso-4380
existing project: kcc-cso-4380
Date: Wed 14 Feb 2024 02:50:28 AM UTC
Timestamp: 1707879028
running with: -b kcc-cso -u cso2 -c false -l false -m false -o false -s true -h false -r false -d false -p kcc-cso-4380
Updated property [core/project].
Switched back to boot project kcc-cso
Start: 1707879029
unique string: cso2
REGION: northamerica-northeast1
NETWORK: kcc-vpc
SUBNET: kcc-sn
CLUSTER: kcc
Reusing project: kcc-cso-4380
CC_PROJECT_ID: kcc-cso-4380
BOOT_PROJECT_ID: kcc-cso
BILLING_ID: 01B35D-D56E1A-BAE17A
ORG_ID: 734065690346
Switching to KCC project kcc-cso-4380
Updated property [core/project].
deploy {REL_SUB_PACKAGE}
KCC_PROJECT_NUMBER: 343139601407
generated derived setters-client-project-setup.yaml
Directory kpt exists - using it
deploying client-project-setup
get kpt release package solutions/client-project-setup version 0.4.6
Package "client-project-setup":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.4.6
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/client-project-setup/0.4.6 -> FETCH_HEAD
Adding package "solutions/client-project-setup".

Fetched 1 package(s).
copy over generated setters.yaml
removing gitops directory
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "client-project-setup": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.6s
  Results:
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-serviceaccountadmin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-securityadmin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-tier3-firewallrule-admin-app-infra-folder-permissions"
    [info] metadata.namespace: set field value to "client1-hierarchy"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.role: set field value to "organizations/734065690346/roles/tier3.firewallrule.admin"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-tier3-dnsrecord-admin-net-host-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "net-host-project-cso3"
    [info] spec.role: set field value to "organizations/734065690346/roles/tier3.dnsrecord.admin"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-compute-public-ip-admin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-compute-security-admin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-workload-identity-binding"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3-tier3-sa"
    [info] spec.resourceRef.namespace: set field value to "client1-config-control"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-client-project-cso3-tier3]"
    [info] metadata.name: set field value to "client-project-cso3-tier3"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.googleServiceAccount: set field value to "tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client1-logging"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier4"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier3"
    [info] subjects[0].name: set field value to "ns-reconciler-client-project-cso3-tier3"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa-networkuser-project-id-nane1-standard-nonp-main-snet-permissions"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "project-id-nane1-standard-nonp-main-snet"
    [info] spec.member: set field value to "serviceAccount:tier4-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa-networkuser-project-id-nane2-standard-nonp-main-snet-permissions"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "project-id-nane2-standard-nonp-main-snet"
    [info] spec.member: set field value to "serviceAccount:tier4-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa-workload-identity-binding"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3-tier4-sa"
    [info] spec.resourceRef.namespace: set field value to "client1-config-control"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-client-project-cso3-tier4]"
    [info] metadata.name: set field value to "client-project-cso3-tier4"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier4"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.googleServiceAccount: set field value to "tier4-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier4"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier4"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier4"
    [info] metadata.namespace: set field value to "client-project-cso3-tier3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier4"
    [info] metadata.namespace: set field value to "client-project-cso3-tier4"
    [info] subjects[0].name: set field value to "ns-reconciler-client-project-cso3-tier4"
    [info] metadata.name: set field value to "client1-config-control-sa-iamserviceaccountadmin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:client1-config-control-sa@client-management-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] spec.name: set field value to "client-project-cso3"
    [info] spec.billingAccountRef.external: set field value to "01B35D-D56E1A-BAE17A"
    [info] spec.folderRef.name: set field value to "standard.applications.nonp"
    [info] spec.folderRef.namespace: set field value to "client1-hierarchy"
    [info] metadata.name: set field value to "client-project-cso3-svpcservice"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "net-host-project-cso3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.name: set field value to "client-project-cso3"
    [info] spec.projectRef.namespace: set field value to "client1-projects"
    [info] metadata.name: set field value to "client-project-cso3-iam"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-resourcemanager"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-billing"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-serviceusage"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-logging"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-monitoring"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply after 60s wait

rerun with

  allowed-nane1-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane2-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane2-standard-${CLIENT_CLASSIFICATION}-main-snet

michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ cd ../../../kpt/
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply client-project-setup --reconcile-timeout=15m --output=table

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render client-project-setup --truncate-output=false
Package "client-project-setup": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
  Results:
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-serviceaccountadmin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-securityadmin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-tier3-firewallrule-admin-app-infra-folder-permissions"
    [info] metadata.namespace: set field value to "client1-hierarchy"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.role: set field value to "organizations/734065690346/roles/tier3.firewallrule.admin"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-tier3-dnsrecord-admin-net-host-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "net-host-project-cso3"
    [info] spec.role: set field value to "organizations/734065690346/roles/tier3.dnsrecord.admin"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-compute-public-ip-admin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-compute-security-admin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier3-sa-workload-identity-binding"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3-tier3-sa"
    [info] spec.resourceRef.namespace: set field value to "client1-config-control"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-client-project-cso3-tier3]"
    [info] metadata.name: set field value to "client-project-cso3-tier3"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.googleServiceAccount: set field value to "tier3-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client1-logging"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier4"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier3"
    [info] subjects[0].name: set field value to "ns-reconciler-client-project-cso3-tier3"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa-networkuser-net-host-project-cso3-nane1-standard-nonp-main-snet-permissions"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "net-host-project-cso3-nane1-standard-nonp-main-snet"
    [info] spec.member: set field value to "serviceAccount:tier4-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa-networkuser-net-host-project-cso3-nane2-standard-nonp-main-snet-permissions"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "net-host-project-cso3-nane2-standard-nonp-main-snet"
    [info] spec.member: set field value to "serviceAccount:tier4-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3-tier4-sa-workload-identity-binding"
    [info] metadata.namespace: set field value to "client1-config-control"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3-tier4-sa"
    [info] spec.resourceRef.namespace: set field value to "client1-config-control"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-cso-4380.svc.id.goog[cnrm-system/cnrm-controller-manager-client-project-cso3-tier4]"
    [info] metadata.name: set field value to "client-project-cso3-tier4"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "client-project-cso3"
    [info] metadata.namespace: set field value to "client-project-cso3-tier4"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.googleServiceAccount: set field value to "tier4-sa@client-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier4"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier4"
    [info] metadata.name: set field value to "cnrm-viewer-client-project-cso3-tier4"
    [info] metadata.namespace: set field value to "client-project-cso3-tier3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] subjects[0].name: set field value to "cnrm-controller-manager-client-project-cso3-tier4"
    [info] metadata.namespace: set field value to "client-project-cso3-tier4"
    [info] subjects[0].name: set field value to "ns-reconciler-client-project-cso3-tier4"
    [info] metadata.name: set field value to "client1-config-control-sa-iamserviceaccountadmin-client-project-cso3-permissions"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.resourceRef.name: set field value to "client-project-cso3"
    [info] spec.member: set field value to "serviceAccount:client1-config-control-sa@client-management-project-cso3.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "client-project-cso3"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] spec.name: set field value to "client-project-cso3"
    [info] spec.billingAccountRef.external: set field value to "01B35D-D56E1A-BAE17A"
    [info] spec.folderRef.name: set field value to "standard.applications.nonp"
    [info] spec.folderRef.namespace: set field value to "client1-hierarchy"
    [info] metadata.name: set field value to "client-project-cso3-svpcservice"
    [info] metadata.namespace: set field value to "client1-networking"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "net-host-project-cso3"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.name: set field value to "client-project-cso3"
    [info] spec.projectRef.namespace: set field value to "client1-projects"
    [info] metadata.name: set field value to "client-project-cso3-iam"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-resourcemanager"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-billing"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-serviceusage"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-logging"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"
    [info] metadata.name: set field value to "client-project-cso3-monitoring"
    [info] metadata.namespace: set field value to "client1-projects"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/client1-projects/Project/client-project-cso3"
    [info] spec.projectRef.external: set field value to "client-project-cso3"

Successfully executed 1 function(s) in 1 package(s)

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/client-project-cso3-tier3       Successful    Current                 <None>                                    12m     Resource is current                     
            Namespace/client-project-cso3-tier4       Successful    Current                 <None>                                    12m     Resource is current                     
client-pro  ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -                                               
client-pro  RoleBinding/cnrm-viewer-client-project-c  Skipped       Unknown                 -                                         -                                               
client-pro  RoleBinding/syncs-repo                    Successful    Current                 <None>                                    12m     Resource is current                     
client-pro  ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -                                               
client-pro  RoleBinding/cnrm-viewer-client-project-c  Skipped       Unknown                 -                                         -                                               
client-pro  RoleBinding/syncs-repo                    Successful    Current                 <None>                                    12m     Resource is current                     
client1-co  IAMPartialPolicy/client-project-cso3-tie  Skipped       Unknown                 -                                         -                                               
client1-co  IAMPartialPolicy/client-project-cso3-tie  Skipped       Unknown                 -                                         -                                               
client1-co  IAMServiceAccount/client-project-cso3-ti  Skipped       Unknown                 -                                         -                                               
client1-co  IAMServiceAccount/client-project-cso3-ti  Skipped       Unknown                 -                                         -                                               
client1-hi  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-lo  RoleBinding/cnrm-viewer-client-project-c  Skipped       Unknown                 -                                         -                                               
client1-ne  ComputeSharedVPCServiceProject/client-pr  Skipped       Unknown                 -                                         -                                               
client1-ne  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-ne  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-ne  RoleBinding/cnrm-viewer-client-project-c  Skipped       Unknown                 -                                         -                                               
client1-ne  RoleBinding/cnrm-viewer-client-project-c  Skipped       Unknown                 -                                         -                                               
client1-pr  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-pr  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-pr  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-pr  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-pr  IAMPolicyMember/client-project-cso3-tier  Skipped       Unknown                 -                                         -                                               
client1-pr  IAMPolicyMember/client1-config-control-s  Skipped       Unknown                 -                                         -                                               
client1-pr  RoleBinding/cnrm-viewer-client-project-c  Skipped       Unknown                 -                                         -                                               
client1-pr  Project/client-project-cso3                             Unknown                 -                                         -                                               
client1-pr  Service/client-project-cso3-billing       Skipped       Unknown                 -                                         -                                               
client1-pr  Service/client-project-cso3-iam           Skipped       Unknown                 -                                         -                                               
client1-pr  Service/client-project-cso3-logging       Skipped       Unknown                 -                                         -                                               
client1-pr  Service/client-project-cso3-monitoring    Skipped       Unknown                 -                                         -                                               
client1-pr  Service/client-project-cso3-resourcemana  Skipped       Unknown                 -                                         -                                               
client1-pr  Service/client-project-cso3-serviceusage  Skipped       Unknown                 -                                         -                

[obriensystems]

subnets should be of the form without project prefix - see #830 nane1-standard-nonp-main-snet

tried [info] spec.folderRef.name: set field value to "202541361947" [info] spec.folderRef.namespace: set field value to "client1-hierarchy"

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n client1-hierarchy No resources found in client1-hierarchy namespace.

needs to be the same as [clients.clientnam](kpt-set: clients.${client-name})

clients.clients-cso3.standard.applications.nonp

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live destroy client-project-setup
delete phase started
rolebinding.rbac.authorization.k8s.io/syncs-repo delete successful
rolebinding.rbac.authorization.k8s.io/syncs-repo delete successful
delete phase finished
reconcile phase started
rolebinding.rbac.authorization.k8s.io/syncs-repo reconcile successful
rolebinding.rbac.authorization.k8s.io/syncs-repo reconcile pending
rolebinding.rbac.authorization.k8s.io/syncs-repo reconcile successful
reconcile phase finished
delete phase started
namespace/client-project-cso3-tier4 delete successful
namespace/client-project-cso3-tier3 delete successful
delete phase finished
reconcile phase started
namespace/client-project-cso3-tier4 reconcile pending
namespace/client-project-cso3-tier3 reconcile pending
namespace/client-project-cso3-tier3 reconcile successful
namespace/client-project-cso3-tier4 reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 4 attempted, 4 successful, 0 skipped, 0 failed
reconcile result: 4 attempted, 4 successful, 0 skipped, 0 failed, 0 timed out

forgot same rootsync folder needs deleting

  echo "removing gitops directory"
  rm -rf $REL_SUB_PACKAGE/root-sync-git

config-man  RootSync/client-project-cso3-t3-csync     Successful    Failed                  Stalled                                   3m      Secret git-creds not found: create one t
config-man  RootSync/client-project-cso3-t4-csync     Successful    Failed                  Stalled                                   3m      Secret git-creds not found: create one t

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status client-project-setup
inventory-69902812/iamserviceaccount.iam.cnrm.cloud.google.com/client1-config-control/client-project-cso3-tier3-sa is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-projects/client-project-cso3-tier3-sa-serviceaccountadmin-client-project-cso3-permissions is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-projects/client-project-cso3-tier3-sa-securityadmin-client-project-cso3-permissions is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-hierarchy/client-project-cso3-tier3-sa-tier3-firewallrule-admin-app-infra-folder-permissions is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-projects/client-project-cso3-tier3-sa-tier3-dnsrecord-admin-net-host-project-cso3-permissions is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-projects/client-project-cso3-tier3-sa-compute-public-ip-admin-client-project-cso3-permissions is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-projects/client-project-cso3-tier3-sa-compute-security-admin-client-project-cso3-permissions is NotFound: Resource not found
inventory-69902812/iampartialpolicy.iam.cnrm.cloud.google.com/client1-config-control/client-project-cso3-tier3-sa-workload-identity-binding is NotFound: Resource not found
inventory-69902812/namespace//client-project-cso3-tier3 is Current: Resource is current
inventory-69902812/configconnectorcontext.core.cnrm.cloud.google.com/client-project-cso3-tier3/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client1-projects/cnrm-viewer-client-project-cso3-tier3 is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client1-networking/cnrm-viewer-client-project-cso3-tier3 is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client1-logging/cnrm-viewer-client-project-cso3-tier3 is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client-project-cso3-tier4/cnrm-viewer-client-project-cso3-tier3 is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client-project-cso3-tier3/syncs-repo is NotFound: Resource not found
inventory-69902812/iamserviceaccount.iam.cnrm.cloud.google.com/client1-config-control/client-project-cso3-tier4-sa is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-networking/client-project-cso3-tier4-sa-networkuser-nane1-standard-nonp-main-snet-permissions is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-networking/client-project-cso3-tier4-sa-networkuser-nane2-standard-nonp-main-snet-permissions is NotFound: Resource not found
inventory-69902812/iampartialpolicy.iam.cnrm.cloud.google.com/client1-config-control/client-project-cso3-tier4-sa-workload-identity-binding is NotFound: Resource not found
inventory-69902812/namespace//client-project-cso3-tier4 is Current: Resource is current
inventory-69902812/configconnectorcontext.core.cnrm.cloud.google.com/client-project-cso3-tier4/configconnectorcontext.core.cnrm.cloud.google.com is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client1-networking/cnrm-viewer-client-project-cso3-tier4 is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client-project-cso3-tier3/cnrm-viewer-client-project-cso3-tier4 is NotFound: Resource not found
inventory-69902812/rolebinding.rbac.authorization.k8s.io/client-project-cso3-tier4/syncs-repo is NotFound: Resource not found
inventory-69902812/iampolicymember.iam.cnrm.cloud.google.com/client1-projects/client1-config-control-sa-iamserviceaccountadmin-client-project-cso3-permissions is NotFound: Resource not found
inventory-69902812/project.resourcemanager.cnrm.cloud.google.com/client1-projects/client-project-cso3 is NotFound: Resource not found
inventory-69902812/computesharedvpcserviceproject.compute.cnrm.cloud.google.com/client1-networking/client-project-cso3-svpcservice is NotFound: Resource not found
inventory-69902812/rootsync.configsync.gke.io/config-management-system/client-project-cso3-t3-csync is Failed: Secret git-creds not found: create one to allow client authentication
inventory-69902812/rootsync.configsync.gke.io/config-management-system/client-project-cso3-t4-csync is Failed: Secret git-creds not found: create one to allow client authentication
inventory-69902812/service.serviceusage.cnrm.cloud.google.com/client1-projects/client-project-cso3-iam is NotFound: Resource not found
inventory-69902812/service.serviceusage.cnrm.cloud.google.com/client1-projects/client-project-cso3-resourcemanager is NotFound: Resource not found
inventory-69902812/service.serviceusage.cnrm.cloud.google.com/client1-projects/client-project-cso3-billing is NotFound: Resource not found
inventory-69902812/service.serviceusage.cnrm.cloud.google.com/client1-projects/client-project-cso3-serviceusage is NotFound: Resource not found
inventory-69902812/service.serviceusage.cnrm.cloud.google.com/client1-projects/client-project-cso3-logging is NotFound: Resource not found
inventory-69902812/service.serviceusage.cnrm.cloud.google.com/client1-projects/client-project-cso3-monitoring is NotFound: Resource not found

[obriensystems]

restart without rootsync

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live destroy client-project-setup
delete phase started
rootsync.configsync.gke.io/client-project-cso3-t4-csync delete successful
rootsync.configsync.gke.io/client-project-cso3-t3-csync delete successful
namespace/client-project-cso3-tier4 delete successful
namespace/client-project-cso3-tier3 delete successful
delete phase finished
reconcile phase started
rootsync.configsync.gke.io/client-project-cso3-t4-csync reconcile successful
rootsync.configsync.gke.io/client-project-cso3-t3-csync reconcile successful
namespace/client-project-cso3-tier4 reconcile pending
namespace/client-project-cso3-tier3 reconcile pending
namespace/client-project-cso3-tier3 reconcile successful
namespace/client-project-cso3-tier4 reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 4 attempted, 4 successful, 0 skipped, 0 failed
reconcile result: 4 attempted, 4 successful, 0 skipped, 0 failed, 0 timed out

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ rm -rf client-project-setup/
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ cd ../github/pubsec-declarative-toolkit/solutions/
michael@cloudshell:~/kcc-cso/github/pubsec-declarative-toolkit/solutions (kcc-cso-4380)$ ./setup.sh -b kcc-cso -u cso2 -n false -c false -l false -m false -o false -s true -g false -h false -r false -d false -j false -p kcc-cso-4380

all except 4 skipped
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status client-project-setup
inventory-73773701/namespace//client-project-cso3-tier3 is Current: Resource is current
inventory-73773701/rolebinding.rbac.authorization.k8s.io/client-project-cso3-tier3/syncs-repo is Current: Resource is current
inventory-73773701/namespace//client-project-cso3-tier4 is Current: Resource is current
inventory-73773701/rolebinding.rbac.authorization.k8s.io/client-project-cso3-tier4/syncs-repo is Current: Resource is current

with
data:
  org-id: "${ORG_ID}"  
  management-project-id: "${KCC_PROJECT_ID}"
  management-namespace: "${MANAGEMENT_NAMESPACE}"
  client-name: '${CLIENT_NAME_1}'
  client-management-project-id: client-management-project-${PREFIX_CLIENT_SETUP}
  host-project-id: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}
  # see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26
  #allowed-nane1-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  #allowed-nane2-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane1-main-subnet: nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane2-main-subnet: nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  project-id: client-project-${PREFIX_CLIENT_PROJECT_SETUP}
  project-billing-id: "${BILLING_ID}"
  project-parent-folder: clients.clients-cso3.standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}
  repo-url: git-repo-to-observe
  repo-branch: main
  tier3-repo-dir: csync/tier3/configcontroller/deploy/env
  tier4-repo-dir: csync/tier4/configcontroller/deploy/env

rendered as
data:
  org-id: "73...46"
  management-project-id: "kcc-cso-4380"
  management-namespace: "config-control"
  client-name: 'client1'
  client-management-project-id: client-management-project-cso3
  host-project-id: net-host-project-cso3
  # see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26
  #allowed-nane1-main-subnet: net-host-project-cso3-nane1-standard-nonp-main-snet
  #allowed-nane2-main-subnet: net-host-project-cso3-nane2-standard-nonp-main-snet
  allowed-nane1-main-subnet: nane1-standard-nonp-main-snet
  allowed-nane2-main-subnet: nane2-standard-nonp-main-snet
  project-id: client-project-cso3
  project-billing-id: "01..17A"
  project-parent-folder: clients.clients-cso3.standard.applications.nonp
  repo-url: git-repo-to-observe
  repo-branch: main
  tier3-repo-dir: csync/tier3/configcontroller/deploy/env
  tier4-repo-dir: csync/tier4/configcontroller/deploy/env

thinking client1 client-name should be client-cso3 - matching the folder
from client-setup
client-name: client-cso3

that seems to be working partially - need to get the folder right

  project-parent-folder: clients.clients-cso3.standard.applications.nonp
inventory-65598980/project.resourcemanager.cnrm.cloud.google.com/client-cso3-projects/client-project-cso3 is InProgress: reference Folder client-cso3-hierarchy/clients.clients-cso3.standard.applications.nonp is not found

extra s - removing it
project-parent-folder: clients.client-${PREFIX_CLIENT_SETUP}.standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}

    inventory-79772528/project.resourcemanager.cnrm.cloud.google.com/client-cso3-projects/client-project-cso3 is InProgress: reference Folder client-cso3-hierarchy/clients.client-cso3.standard.applications.nonp is not found
  folderRef:
    name: clients.client-cso3.standard.applications.nonp # kpt-set: ${project-parent-folder}
    namespace: client-cso3-hierarchy # kpt-set: ${client-name}-hierarchy

   it looks like the namespace is wrong - the folder reference is not in
inventory-15023513/project.resourcemanager.cnrm.cloud.google.com/client-cso3-projects/client-project-cso3 is InProgress: reference Folder client-cso3-hierarchy/clients.client-cso3.standard.applications-infrastructure.nonp is not found

we are there though
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get gcp -n client-cso3-hierarchy
NAME                                                                                             AGE     READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/clients.client-cso3-client-folder-viewer-permissions   7d10h   True    UpToDate   7d10h

NAME                                                                                     AGE     READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/standard                                    7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.applications                       7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.applications-infrastructure        7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.applications-infrastructure.nonp   7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.applications-infrastructure.pbmm   7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.applications.nonp                  7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.applications.pbmm                  7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.auto                               7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.auto.nonp                          7d10h   True    UpToDate   7d10h
folder.resourcemanager.cnrm.cloud.google.com/standard.auto.pbmm                          7d10h   True    UpToDate   7d10h

switch to subfolder
  project-parent-folder: standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}

we are good

[obriensystems]

client-project-setup

data:
  org-id: "${ORG_ID}"  
  management-project-id: "${KCC_PROJECT_ID}"
  management-namespace: "${MANAGEMENT_NAMESPACE}"
  client-name: client-${PREFIX_CLIENT_SETUP}
  client-management-project-id: client-management-project-${PREFIX_CLIENT_SETUP}
  host-project-id: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}
  # see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26
  #allowed-nane1-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  #allowed-nane2-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane1-main-subnet: nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane2-main-subnet: nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  project-id: client-project-${PREFIX_CLIENT_PROJECT_SETUP}
  project-billing-id: "${BILLING_ID}"
#  project-parent-folder: clients.client-${PREFIX_CLIENT_SETUP}.standard.applications-infrastructure.${CLIENT_PROJECT_PARENT_FOLDER}
  project-parent-folder: standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}
  repo-url: git-repo-to-observe
  repo-branch: main
  tier3-repo-dir: csync/tier3/configcontroller/deploy/env
  tier4-repo-dir: csync/tier4/configcontroller/deploy/env

[fmichaelobrien]

add to setup.sh - Anoop's RBAC addition in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/834

[obriensystems]

update LZ with package updates spawn https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/835

 * [new tag]         solutions/client-landing-zone/0.5.4                       -> solutions/client-landing-zone/0.5.4
 * [new tag]         solutions/client-project-setup/0.4.6                      -> solutions/client-project-setup/0.4.6
 * [new tag]         solutions/client-setup/0.7.2                              -> solutions/client-setup/0.7.2
 * [new tag]         solutions/core-landing-zone/0.7.2                         -> solutions/core-landing-zone/0.7.2

[obriensystems]

Retesting hub-env automation Note: previous live cluster is on up r @ ls review: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/samples/Egress-workload-to-internet.md

./setup.sh -b kcc-cso -u cso2 -n false -c false -l false -m false -o false -s false -g false -h true -r false -d false -j false -p kcc-cso-4380

get kpt release package solutions/project/hub-env version 0.2.2

forgot billing_id change
  project-billing-id: ""

ichael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live destroy hub-env
delete phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-cso2 delete successful
delete phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-cso2 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-cso2 reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 1 attempted, 1 successful, 0 skipped, 0 failed
reconcile result: 1 attempted, 1 successful, 0 skipped, 0 failed, 0 timed out

redeploy
projects    Project/xxdmu-admin1-hub-cso2             Successful    Failed                  Ready                                     6s      Update call failed: error applying desir
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env
inventory-33495783/project.resourcemanager.cnrm.cloud.google.com/projects/xxdmu-admin1-hub-cso2 is Failed: Update call failed: error applying desired state: summary: failed pre-requisites: missing permission on "billingAccounts/01DB...54": billing.resourceAssociations.create

BAU needed on the SA projects-sa@kcc-cso-4380.iam.gserviceaccount.com

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live destroy hub-env
redploy
better

project up

instances up in 5 min

fgt-primary-instance | northamerica-northeast1-a |   |   | 172.31.200.10 (nic0) 172.31.201.10 (nic1) 172.31.203.10 (nic2) 172.31.202.10 (nic3) |   | SSH |   |     |   | fgt-secondary-instance | northamerica-northeast1-b |   |   | 172.31.200.11 (nic0) 172.31.201.11 (nic1) 172.31.203.11 (nic2) 172.31.202.11 (nic3) |   | SSH |   |     |   | management-instance | northamerica-northeast1-a |   |   | 172.31.202.2 (nic0) |   | RDP |   |  

fgt-primary-instance northamerica-northeast1-a
172.31.200.10 (nic0) 172.31.201.10 (nic1) 172.31.203.10 (nic2) 172.31.202.10 (nic3)
fgt-secondary-instance northamerica-northeast1-b
172.31.200.11 (nic0) 172.31.201.11 (nic1) 172.31.203.11 (nic2) 172.31.202.11 (nic3)
management-instance northamerica-northeast1-a
172.31.202.2 (nic0)

[obriensystems]

hub-env status

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Current                 Ready                                     3m      Resource is Current                     
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     3m      Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     3m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Current                 Ready                                     3m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Current                 Ready                                     3m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-ext-addre  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-int-addre  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-transit-a  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-ext-add  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-int-add  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-transit  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-ilb-address            Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeAddress/hub-ilb-proxy-address      Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeBackendService/hub-ilb-bes         Successful    Current                 Ready                                     58s     Resource is Current                     
networking  ComputeDisk/hub-fgt-primary-log-disk      Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeDisk/hub-fgt-secondary-log-disk    Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeDisk/hub-mgmt-data-disk            Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeFirewall/hub-allow-external-fwr    Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeFirewall/hub-allow-fortigates-ha-  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeFirewall/hub-allow-spokes-to-fort  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeFirewall/hub-elb-allow-health-che  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeFirewall/hub-iap-allow-rdp-to-man  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeFirewall/hub-ilb-allow-health-che  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeFirewall/hub-managementvm-allow-s  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeForwardingRule/hub-ilb-fwdrule     Successful    Current                 Ready                                     36s     Resource is Current                     
networking  ComputeForwardingRule/hub-ilb-proxy-fwdr  Successful    Current                 Ready                                     36s     Resource is Current                     
networking  ComputeHTTPHealthCheck/hub-http-8008-htt  Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeHealthCheck/hub-http-8008-hc       Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeInstance/hub-fgt-primary-instance  Successful    Current                 Ready                                     84s     Resource is Current                     
networking  ComputeInstance/hub-fgt-secondary-instan  Successful    Current                 Ready                                     84s     Resource is Current                     
networking  ComputeInstance/hub-management-instance   Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeInstanceGroup/hub-fgt-primary-umi  Successful    Current                 Ready                                     71s     Resource is Current                     
networking  ComputeInstanceGroup/hub-fgt-secondary-u  Successful    Current                 Ready                                     71s     Resource is Current                     
networking  ComputeNetwork/hub-global-external-vpc    Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeNetwork/hub-global-internal-vpc    Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeNetwork/hub-global-mgmt-vpc        Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeNetwork/hub-global-transit-vpc     Successful    Current                 Ready                                     3m      Resource is Current                     
networking  ComputeRoute/hub-external-vpc-internet-e  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeRoute/hub-internal-vpc-internet-e  Successful    Current                 Ready                                     13s     Resource is Current                     
networking  ComputeRouter/hub-nane1-external-router   Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeRouterNAT/hub-nane1-external-nat   Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
networking  ComputeSubnetwork/hub-nane1-external-paz  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-internal-paz  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-mgmt-rz-snet  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-transit-paz-  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  ComputeTargetPool/hub-elb-pool            Successful    Current                 Ready                                     70s     Resource is Current                     
networking  DNSPolicy/hub-external-logging-dnspolicy  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  DNSPolicy/hub-internal-logging-dnspolicy  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  DNSPolicy/hub-mgmt-logging-dnspolicy      Successful    Current                 Ready                                     2m      Resource is Current                     
networking  DNSPolicy/hub-transit-logging-dnspolicy   Successful    Current                 Ready                                     2m      Resource is Current                     
networking  IAMPolicyMember/hub-admin-serviceaccount  Successful    Current                 Ready                                     2m      Resource is Current                     
networking  IAMServiceAccount/hub-fortigatesdn-sa     Successful    Current                 Ready                                     3m      Resource is Current                     
networking  IAMServiceAccount/hub-managementvm-sa     Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     3m      Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    Project/xxdmu-admin1-hub-cso2             Successful    Current                 Ready                                     4m      Resource is Current                     
projects    Service/xxdmu-admin1-hub-cso2-compute     Successful    Current                 Ready                                     3m      Resource is Current                     
projects    Service/xxdmu-admin1-hub-cso2-dns         Successful    Current                 Ready                                     3m      Resource is Current                     

single failure on the nat

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet

[obriensystems]

raised separate https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/837

[obriensystems]

Fixing permissions issue on compute for client project see #838

[obriensystems]

Moving billing for hub-env project to another billing id projects-sa@kcc-cso-4380.iam.gserviceaccount.com

[obriensystems]

Nat issue fixed by adding a restrictCloudNATUsage project level override for hub-env in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/837

[obriensystems]

peering is required see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/847 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/842

[obriensystems]

verify alternate ip address on elb/ilb - for routing to work

check sdnconnector or ha

remember sync ilb custom config int hub-env check workload east-west between two workloads should go through the ilb - policy based routing between services project

research vdom - one also for management vm - which interface to update from fortinet updates from fortinet via public ip? validate if private ips are dhcp - which they are - check fortigate fdynamic-gw settings on interface vDOM

real locked license - sharable?

plan: 5 + separate hub-env pipelines core-landing-zone - org/org-policies global replace from org to project or folderRef

[fmichaelobrien]

git merge main Merge https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/846/files into the gh766-script branch