Add CD (Continuous Deployment/Delivery) build step/script for each package release version - to verify for example core-landing-zone deploys properly

fmichaelobrien commented 7 months ago

Requirement: a full/partial CD to verify a package against an existing/new KCC GKE cluster

Here is a prototype currently running against 20231206 v0.7.0 (derived) - core-landing-zone on a clean KCC GKE cluster
preclean the org
do not deploy the gatekeeper policies for now
see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/654
We periodically run into issues (sometimes with main) where a package has deployment issues that need to be triaged via kubectl describe - see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#deployment-determine-kubernetes-gcp-service-status
see for example https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/752

fmichaelobrien commented 7 months ago

create cluster

oi org running main 20231206:0600 to verify no issue with the following https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/services-infrastructure/folder-sink.yaml#L23 platform-and-component-services-infra-log-sink from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/audits/logging-project/project-iam.yaml#L105 mgmt-project-cluster-platform-and-component-log-sink

michael@cloudshell:~/kcc-oi$ cd ..
michael@cloudshell:~$ mkdir kcc-oi-20231206
michael@cloudshell:~$ cd kcc-oi-20231206/
michael@cloudshell:~/kcc-oi-20231206$ mkdir github
michael@cloudshell:~/kcc-oi-20231206$ mkdir kpt
michael@cloudshell:~/kcc-oi-20231206$ cd github/
michael@cloudshell:~/kcc-oi-20231206/github$ git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git
Cloning into 'pubsec-declarative-toolkit'...
remote: Enumerating objects: 9668, done.
remote: Counting objects: 100% (3488/3488), done.
remote: Compressing objects: 100% (1134/1134), done.
remote: Total 9668 (delta 2779), reused 2785 (delta 2338), pack-reused 6180
Receiving objects: 100% (9668/9668), 6.40 MiB | 12.24 MiB/s, done.
Resolving deltas: 100% (6225/6225), done.
michael@cloudshell:~/kcc-oi-20231206/github$ cd pubsec-declarative-toolkit/
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit$ ls
CONTRIBUTING.md  docs  examples  LICENSE  README.md  release-please-config.json  services  solutions
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit$ cd ..
michael@cloudshell:~/kcc-oi-20231206/github$ mkdir _pull_20231206_0641
michael@cloudshell:~/kcc-oi-20231206/github$ cd pubsec-declarative-toolkit/
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit$ cd ..
michael@cloudshell:~/kcc-oi-20231206/github$ ls
pubsec-declarative-toolkit  _pull_20231206_0641
michael@cloudshell:~/kcc-oi-20231206/github$ mkdir _446-hub
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub$ git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git     
Cloning into 'pubsec-declarative-toolkit'...
remote: Enumerating objects: 9668, done.
remote: Counting objects: 100% (3563/3563), done.
remote: Compressing objects: 100% (1112/1112), done.
remote: Total 9668 (delta 2852), reused 2881 (delta 2435), pack-reused 6105
Receiving objects: 100% (9668/9668), 6.41 MiB | 26.91 MiB/s, done.
Resolving deltas: 100% (6228/6228), done.
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub$ cd pubsec-declarative-toolkit/
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub/pubsec-declarative-toolkit$ git checkout gh446-hub
Branch 'gh446-hub' set up to track remote branch 'gh446-hub' from 'origin'.
Switched to a new branch 'gh446-hub'
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub/pubsec-declarative-toolkit$ cp solutions/
client-landing-zone/  core-landing-zone/    gke/                  ids/                  project/              vars.sh
client-project-setup/ experimentation/      guardrails/           kcc-namespaces/       setup.sh              vertexai/
client-setup/         gatekeeper-policies/  guardrails-policies/  legacy/               solutions.yaml        
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub/pubsec-declarative-toolkit$ cp solutions/vars.sh ../../
_446-hub/                   pubsec-declarative-toolkit/ _pull_20231206_0641/        
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub/pubsec-declarative-toolkit$ cp solutions/vars.sh ../../pubsec-declarative-toolkit/solutions/
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub/pubsec-declarative-toolkit$ cp solutions/setup.sh ../../pubsec-declarative-toolkit/so
lutions/
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub/pubsec-declarative-toolkit$ cd ..
michael@cloudshell:~/kcc-oi-20231206/github/_446-hub$ cd ..
michael@cloudshell:~/kcc-oi-20231206/github$ 

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ chmod 777 setup.sh
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u ar -n true -c true -l false -h false -d false -j false

generated kcc project_id propagation to the end in yakima/sa role additions retested in #654

0648
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u ar -n true -c true -l false -h false -d false -j false
existing project: 
Date: Wed 06 Dec 2023 11:48:39 AM UTC
Timestamp: 1701863319
running with: -b kcc-oi -u ar -c true -l false -h false -r false -d false -p 
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1701863320
unique string: ar
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Creating project: kcc-oi-6475
CC_PROJECT_ID: kcc-oi-6475
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459..44
applying roles to the super admin SUPER_ADMIN_EMAIL: michael@obrien.industries
Updated IAM policy for organization [4..44].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..].
Updated IAM policy for organization [4.4]..
Updated IAM policy for organization [4..144].
Updated IAM policy for organization [459..44].
Creating KCC project: kcc-oi-6475 on folder: 38862..43
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-oi-6475].
Waiting for [operations/cp.5638443903817105010] to finish...done.                                                                                           
Enabling service [cloudapis.googleapis.com] on project [kcc-oi-6475]...
Operation "operations/acat.p2-993154031891-29201c86-a034-44cc-a146-92e3e696b676" finished successfully.
Updated property [core/project] to [kcc-oi-6475].
Updated property [core/project].
Enabling billing on account: 014..85
billingAccountName: billingAccounts/014..5
billingEnabled: true
name: projects/kcc-oi-6475/billingInfo
projectId: kcc-oi-6475
sleep 45 sec before enabling services
Enabling APIs
Operation "operations/acf.p2-993154031891-7d0764e3-2cd3-49e7-8fb3-102ebcc9c323" finished successfully.
Operation "operations/acat.p2-993154031891-d64f4422-74fd-48c8-a84b-c664d443bb03" finished successfully.
Operation "operations/acat.p2-993154031891-512f8af5-90e8-42e4-8ec0-5b6ad758cf31" finished successfully.
Operation "operations/acat.p2-993154031891-cf30917a-8316-439f-b3c4-67035ae22681" finished successfully.
Operation "operations/acat.p2-993154031891-de537f80-1838-463a-991e-5dfb9fbcd191" finished successfully.
Operation "operations/acat.p2-993154031891-fc32e1ef-6444-4b10-af5a-73a29e981b21" finished successfully.
name: organizations/459065442144/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-6475/global/networks/kcc-ls-vpc].
NAME: kcc-ls-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp:22,tcp:3389,icmp

Create subnet kcc-ls-sn off VPC: kcc-ls-vpc using 192.168.0.0/16 on region: northamerica-northeast1
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-6475/regions/northamerica-northeast1/subnetworks/kcc-ls-sn].
NAME: kcc-ls-sn
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
create default firewalls
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc on project kcc-oi-6475
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-6475/locations/northamerica-northeast1/operations/operation-1701863484478-60bd5f872d47b-3203efc2-fe59f81d] to complet
e...working        

Waiting for operation [projects/kcc-oi-6475/locations/northamerica-northeast1/operations/operation-1701863484478-60bd5f872d47b-3203efc2-fe59f81d] to complet
e...working.                                                                                                                                                
e...working.                                                                                                                                                
e...working..                                                                                                                                               
e...done.                                                                                                                                                   
Created instance [kcc].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
Cluster create time: 1107 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
List Clusters:
NAME: kcc
LOCATION: northamerica-northeast1
STATE: RUNNING
post GKE cluster create - applying 2 roles to org: 459065442144 and project: kcc-oi-6475 on the yakima gke service account to prep for kpt deployment: service-993154031891@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Updated IAM policy for project [kcc-oi-6475].
Updated IAM policy for organization [459065442144].
Updated IAM policy for organization [459065442144].
Total Duration: 1282 sec
Date: Wed 06 Dec 2023 12:10:02 PM UTC
Timestamp: 1701864602
Updated property [core/project].
Switched back to boot project kcc-oi
**** Done ****
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$    
711

fmichaelobrien commented 7 months ago

deploy core-landing-zone - latest release

deploying core-landing-zone using derived 0.7.0 release tag (not main)

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L285

generated setters.yaml

apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
data:
  org-id: "459..44"
  lz-folder-id: "388627537443"
  billing-id: "014...85"
  management-project-id: "kcc-oi-6475"
  management-project-number: "993154031891"
  management-namespace: config-control
  allowed-trusted-image-projects: |
    - "projects/cos-cloud"
  allowed-contact-domains: |
    - "@obri...tries"
  allowed-policy-domain-members: |
    - "C..kc"
  allowed-vpc-peering: |
    - "under:organizations/45...4"
  logging-project-id: logging-project-oi1206
  security-log-bucket: security-log-bucket-oi1206
  platform-and-component-log-bucket: platform-and-component-log-bucket-oi1206
  retention-locking-policy: "false"
  retention-in-days: "1"
  dns-project-id: dns-project-oi1206
  dns-name: "obr..ies."

  REL_URL="https://raw.githubusercontent.com/GoogleCloudPlatform/pubsec-declarative-toolkit/main/.release-please-manifest.json"
  # check for existing landing-zone
  echo "deploying ${REL_SUB_PACKAGE}"
  REL_VERSION=$(curl -s $REL_URL | jq -r ".\"$REL_PACKAGE\"")
  echo "get kpt release package $REL_PACKAGE version $REL_VERSION"
  rm -rf $REL_SUB_PACKAGE
  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${REL_PACKAGE}@${REL_VERSION}

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u ar -n false -c false -l true -h false -d false -j false -p kcc-oi-6475

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u ar -n false -c false -l true -h false -d false -j false -p kcc-oi-6475
existing project: kcc-oi-6475
Date: Wed 06 Dec 2023 12:18:04 PM UTC
Timestamp: 1701865084
running with: -b kcc-oi -u ar -c false -l true -h false -r false -d false -p kcc-oi-6475
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1701865085
unique string: ar
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Reusing project: kcc-oi-6475
CC_PROJECT_ID: kcc-oi-6475
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459065442144
Switching to KCC project kcc-oi-6475
Updated property [core/project].
wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER: 993154031891
DIRECTORY_CUSTOMER_ID: C03kdhrkc
generated derived setters-core-landing-zone.yaml
Directory kpt exists - using it
deploying core-landing-zone
get kpt release package solutions/core-landing-zone version 0.7.0
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.0
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.7.0 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
copy over generated setters.yaml
removing org/org-policies folder
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.5s
  Results:
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.projectRef.name: set field value to "logging-project-oi1206"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.name: set field value to "platform-and-component-log-bucket-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.projectRef.name: set field value to "logging-project-oi1206"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "logging-project-oi1206"
    [info] metadata.name: set field value to "kcc-oi-6475"
    [info] spec.metricsScope: set field value to "location/global/metricsScopes/logging-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.resourceRef.name: set field value to "logging-project-oi1206"
    [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "org-log-sink-security-logging-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.resourceRef.name: set field value to "logging-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.resourceRef.name: set field value to "logging-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.resourceRef.name: set field value to "logging-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.resourceRef.name: set field value to "logging-project-oi1206"
    [info] metadata.name: set field value to "logging-project-oi1206-data-access-sink"
    [info] spec.projectRef.name: set field value to "logging-project-oi1206"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi1206/locations/northamerica-northeast1/buckets/security-log-bucket"
    [info] metadata.name: set field value to "logging-project-oi1206"
    [info] spec.name: set field value to "logging-project-oi1206"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] metadata.name: set field value to "logging-project-oi1206-logging"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.projectRef.external: set field value to "logging-project-oi1206"
    [info] metadata.name: set field value to "logging-project-oi1206-monitoring"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206"
    [info] spec.projectRef.external: set field value to "logging-project-oi1206"
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi1206"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi1206/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi1206"
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.name: set field value to "dns-project-oi1206-standard-core-public-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi1206"
    [info] spec.dnsName: set field value to "obrien.industries."
    [info] metadata.name: set field value to "dns-project-oi1206"
    [info] spec.name: set field value to "dns-project-oi1206"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] metadata.name: set field value to "dns-project-oi1206-dns"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi1206"
    [info] spec.projectRef.external: set field value to "dns-project-oi1206"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi1206"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi1206/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi1206"
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.name: set field value to "compute-disable-serial-port-logging-except-kcc-oi-6475"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.name: set field value to "compute-require-shielded-vm-except-kcc-oi-6475"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.name: set field value to "compute-restrict-cloud-nat-usage-except-kcc-oi-6475"
    [info] spec.listPolicy.allow.values[0]: set field value to "under:projects/kcc-oi-6475"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi1206"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi1206/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi1206"
    [info] metadata.name: set field value to "kcc-oi-6475-cloudbilling"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.name: set field value to "kcc-oi-6475-cloudresourcemanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.name: set field value to "kcc-oi-6475-serviceusage"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.name: set field value to "kcc-oi-6475-accesscontextmanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.name: set field value to "kcc-oi-6475-anthos"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "kcc-oi-6475"
    [info] spec.member: set field value to "serviceAccount:config-mgmt-mon-default-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[config-management-monitoring/default]"
    [info] spec.googleServiceAccount: set field value to "config-mgmt-mon-default-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "kcc-oi-6475"
    [info] spec.member: set field value to "serviceAccount:gatekeeper-admin-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[gatekeeper-system/gatekeeper-admin]"
    [info] spec.googleServiceAccount: set field value to "gatekeeper-admin-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:hierarchy-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]"
    [info] spec.googleServiceAccount: set field value to "hierarchy-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-monitoring-admin-kcc-oi-6475-permissions"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.resourceRef.external: set field value to "kcc-oi-6475"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-monitoring-admin-logging-project-oi1206-permissions"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "logging-project-oi1206"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-storageadmin-logging-project-oi1206-permissions"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "logging-project-oi1206"
    [info] spec.resourceRef.name: set field value to "logging-project-oi1206"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]"
    [info] spec.googleServiceAccount: set field value to "logging-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:service-993154031891@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "kcc-oi-6475"
    [info] spec.member: set field value to "serviceAccount:service-993154031891@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "kcc-oi-6475"
    [info] spec.member: set field value to "serviceAccount:service-993154031891@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]"
    [info] spec.googleServiceAccount: set field value to "networking-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:policies-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]"
    [info] spec.googleServiceAccount: set field value to "policies-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-6475"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-6475.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]"
    [info] spec.googleServiceAccount: set field value to "projects-sa@kcc-oi-6475.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.name: set field value to "org-log-sink-security-logging-project-oi1206"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi1206/locations/northamerica-northeast1/buckets/security-log-bucket"
    [info] metadata.name: set field value to "org-log-sink-data-access-logging-project-oi1206"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi1206/locations/northamerica-northeast1/buckets/security-log-bucket"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-6475-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-accesscontextmanag

service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-anthos apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudbilling apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudresourcemanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-serviceusage apply successful
apply phase finished
reconcile phase started
namespace/hierarchy reconcile successful
namespace/logging reconcile successful
namespace/networking reconcile successful
namespace/policies reconcile successful
namespace/projects reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-6475-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-accesscontextmanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-anthos reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudbilling reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudresourcemanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-serviceusage reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudbilling reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-accesscontextmanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudresourcemanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-anthos reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-serviceusage reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-6475-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful

0719
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile successful

0721
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile successful
reconcile phase finished
apply phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi1206-permissions apply successful
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-6475 apply successful
folder.resourcemanager.cnrm.cloud.google.com/audits apply successful
folder.resourcemanager.cnrm.cloud.google.com/clients apply successful
folder.resourcemanager.cnrm.cloud.google.com/services apply successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure apply successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi1206 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-6475 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-6475 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-6475 apply successful
apply phase finished
reconcile phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi1206-permissions reconcile pending
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-6475 reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi1206 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-6475 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-6475 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-6475 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-6475 reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-6475 reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-6475 reconcile successful
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-6475 reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile successful

iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions reconcile successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi1206 reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi1206-permissions reconcile successful
reconcile phase finished
apply phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions apply successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi1206 apply successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket apply successful
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi1206 apply successful
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-logging apply successful
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-monitoring apply successful
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket apply successful
apply phase finished
reconcile phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi1206 reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket reconcile pending
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi1206 reconcile pending
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-logging reconcile pending
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-monitoring reconcile pending
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket reconcile pending
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi1206 reconcile successful
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile successful
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-monitoring reconcile successful
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-logging reconcile successful
0727
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi1206 reconcile successful

0750

obriensystems commented 7 months ago

Triage any service deployment issues

750 ctrl-c - reduce reconcile timeout

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   29m   True    UpToDate   29m

NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29m   False   DependencyNotFound   29m

NAME                                                                                                       AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions   32m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi1206-permissions       32m   True    UpToDate   29m

NAME                                                                   AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi1206       29m   True    UpToDate   28m
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi1206   32m   True    UpToDate   29m

NAME                                                                           AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-logging      29m   True    UpToDate   29m
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-monitoring   29m   True    UpToDate   29m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n networking
No resources found in networking namespace.
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$   kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    33m   True    UpToDate   31m
folder.resourcemanager.cnrm.cloud.google.com/clients                   33m   True    UpToDate   31m
folder.resourcemanager.cnrm.cloud.google.com/services                  33m   True    UpToDate   31m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   32m   True    UpToDate   31m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$  kubectl get gcp -n policies
NAME                                                                                                                 AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-6475   33m   True    UpToDate   32m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-6475           33m   True    UpToDate   32m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-6475      33m   True    UpToDate   32m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n logging
NAME                                                                                      AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi1206   31m   True    UpToDate   31m
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                        31m   True    UpToDate   31m

NAME                                                                      AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-6475   33m   True    UpToDate   30m

NAME                                                                       AGE   READY   STATUS     STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket   31m   True    UpToDate   31m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n config-control
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   35m   True    UpToDate   35m

NAME                                                                                              AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding   35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding          35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding                 35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding                   35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding                35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding                  35m   True    UpToDate   34m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding                  35m   True    UpToDate   34m

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions                   35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-6475-permissions                    35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 35m   True    UpToDate   34m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    35m   True    UpToDate   34m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 35m   True    UpToDate   34m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 35m   True    UpToDate   33m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                35m   True    UpToDate   33m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   35m   True    UpToDate   33m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              35m   True    UpToDate   33m

NAME                                                                     AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa   35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa          35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa                 35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa                   35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa                35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa                  35m   True    UpToDate   34m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa                  35m   True    UpToDate   34m

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-accesscontextmanager   35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-anthos                 35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudbilling           35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudresourcemanager   35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-serviceusage           35m   True    UpToDate   35m

Issues with

iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29m   False   DependencyNotFound   29m

kubens config-control
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp | grep UpdateFailed
(not representative)

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions -n projects
Name:         security-log-bucket-writer-permissions
Namespace:    projects
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              config.k8s.io/owning-inventory: ace11e1affe3760bdf91752781e6fec950f9ba61-1701865095377759288
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206
              internal.kpt.dev/upstream-identifier: iam.cnrm.cloud.google.com|IAMPartialPolicy|projects|security-log-bucket-writer-permissions
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMPartialPolicy
Metadata:
  Creation Timestamp:  2023-12-06T12:22:34Z
  Generation:          1
  Resource Version:    25635
  UID:                 234ca697-ddd8-43a1-a3ed-e74f5a51d002
Spec:
  Bindings:
    Members:
      Member From:
        Log Sink Ref:
          Name:       org-log-sink-security-logging-project-oi1206
          Namespace:  logging
    Role:             roles/logging.bucketWriter
  Resource Ref:
    API Version:  resourcemanager.cnrm.cloud.google.com/v1beta1
    Kind:         Project
    Name:         logging-project-oi1206
    Namespace:    projects
Status:
  Conditions:
    Last Transition Time:  2023-12-06T12:22:35Z
    Message:               reference LoggingLogSink logging/org-log-sink-security-logging-project-oi1206 is not found
    Reason:                DependencyNotFound
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  DependencyNotFound  5m12s (x4 over 33m)  iampartialpolicy-controller  reference LoggingLogSink logging/org-log-sink-security-logging-project-oi1206 is not found
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ 

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions -n projects
Name:         mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
Namespace:    projects
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              config.k8s.io/owning-inventory: ace11e1affe3760bdf91752781e6fec950f9ba61-1701865095377759288
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206
              internal.kpt.dev/upstream-identifier:
                iam.cnrm.cloud.google.com|IAMPartialPolicy|projects|mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMPartialPolicy
Metadata:
  Creation Timestamp:  2023-12-06T12:22:33Z
  Generation:          1
  Resource Version:    25600
  UID:                 bafe8b5a-8d98-466c-b76a-5e5f53ab509b
Spec:
  Bindings:
    Members:
      Member From:
        Log Sink Ref:
          Name:       mgmt-project-cluster-platform-and-component-log-sink
          Namespace:  logging
    Role:             roles/logging.bucketWriter
  Resource Ref:
    API Version:  resourcemanager.cnrm.cloud.google.com/v1beta1
    Kind:         Project
    Name:         logging-project-oi1206
    Namespace:    projects
Status:
  Conditions:
    Last Transition Time:  2023-12-06T12:22:34Z
    Message:               reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
    Reason:                DependencyNotFound
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  DependencyNotFound  103s (x5 over 37m)  iampartialpolicy-controller  reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found

Triage up the chain

Warning  DependencyNotFound  5m12s (x4 over 33m)  iampartialpolicy-controller  reference LoggingLogSink logging/org-log-sink-security-logging-project-oi1206 is not found

  Warning  DependencyNotFound  103s (x5 over 37m)  iampartialpolicy-controller  reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found

running main 20231206:0600 to verify no issue with the following https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/services-infrastructure/folder-sink.yaml#L23 platform-and-component-services-infra-log-sink from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/audits/logging-project/project-iam.yaml#L105 mgmt-project-cluster-platform-and-component-log-sink

fmichaelobrien commented 7 months ago

raised https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/752

obriensystems commented 7 months ago


step 5
michael@cloudshell:~ (kcc-oi-6475)$ gcloud beta billing accounts add-iam-policy-binding "0..5" --member "serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com" --role "roles/billing.user"
Updated IAM policy for account [01..85].
bindings:
- members:
  - user:michael@obrien.industries
  - user:michael@obrien.software
  - user:michael@obrienlabs.dev
  - user:root@kcc.landing.systems
  role: roles/billing.admin
- members:
  - serviceAccount:projects-sa@kcc-oi-6475.iam.gserviceaccount.com
  role: roles/billing.user
etag: BwYL2EFL9LQ=
version: 1

ichael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-6475)$ kpt live apply core-landing-zone
installing inventory ResourceGroup CRD.

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-6475)$ kubectl get gcp -n projects
NAME                                                                              AGE    READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   139m   True    UpToDate   139m

NAME                                                                                                                   AGE    READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   139m   False   DependencyNotFound   139m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         139m   False   DependencyNotFound   139m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               139m   False   DependencyNotFound   139m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      139m   False   DependencyNotFound   139m

NAME                                                                                                       AGE    READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions   141m   True    UpToDate   140m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi1206-permissions       141m   True    UpToDate   139m

NAME                                                                   AGE    READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi1206       139m   True    UpToDate   138m
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi1206   141m   True    UpToDate   139m

NAME                                                                           AGE    READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-logging      139m   True    UpToDate   139m
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-monitoring   139m   True    UpToDate   139m

fmichaelobrien commented 7 months ago

Summary:for shared billing accounts the projects-sa KCC focused SA must be associated as a BAU for direct accounts we are okhttps://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#5-perform-the-post-deployment-steps https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L332

without this the projects don't get created for shared billing users - and the log syncs as expected won't render until the project is up

direct billing associates the SA with the BAU ok

GoogleCloudPlatform / pubsec-declarative-toolkit