search

GoogleCloudPlatform / pubsec-declarative-toolkit

The GCP PubSec Declarative Toolkit is a collection of declarative solutions to help you on your Journey to Google Cloud. Solutions are designed using Config Connector and deployed using Config Controller.
Apache License 2.0
30 stars 27 forks source link

Regression: core-landing-zone 0.7.0 (20231206) has kpt-set naming issues with mgmt-project-cluster-platform-and-component-log-sink #752

Closed fmichaelobrien closed 5 months ago

fmichaelobrien commented 7 months ago

see #799

See automated KCC cluster creation with auto core-landing-zone (latest release not main)

751

Triage any service deployment issues

We periodically run into issues (sometimes with main) where a package has deployment issues that need to be triaged via kubectl describe - see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#deployment-determine-kubernetes-gcp-service-status

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   29m   True    UpToDate   29m

NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29m   False   DependencyNotFound   29m

NAME                                                                                                       AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi1206-permissions   32m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi1206-permissions       32m   True    UpToDate   29m

NAME                                                                   AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi1206       29m   True    UpToDate   28m
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi1206   32m   True    UpToDate   29m

NAME                                                                           AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-logging      29m   True    UpToDate   29m
service.serviceusage.cnrm.cloud.google.com/logging-project-oi1206-monitoring   29m   True    UpToDate   29m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n networking
No resources found in networking namespace.
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$   kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    33m   True    UpToDate   31m
folder.resourcemanager.cnrm.cloud.google.com/clients                   33m   True    UpToDate   31m
folder.resourcemanager.cnrm.cloud.google.com/services                  33m   True    UpToDate   31m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   32m   True    UpToDate   31m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$  kubectl get gcp -n policies
NAME                                                                                                                 AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-6475   33m   True    UpToDate   32m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-6475           33m   True    UpToDate   32m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-6475      33m   True    UpToDate   32m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n logging
NAME                                                                                      AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi1206   31m   True    UpToDate   31m
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                        31m   True    UpToDate   31m

NAME                                                                      AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-6475   33m   True    UpToDate   30m

NAME                                                                       AGE   READY   STATUS     STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket   31m   True    UpToDate   31m
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp -n config-control
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           35m   True    UpToDate   35m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   35m   True    UpToDate   35m

NAME                                                                                              AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding   35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding          35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding                 35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding                   35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding                35m   True    UpToDate   35m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding                  35m   True    UpToDate   34m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding                  35m   True    UpToDate   34m

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions                   35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-6475-permissions                    35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     35m   True    UpToDate   35m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 35m   True    UpToDate   34m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    35m   True    UpToDate   34m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 35m   True    UpToDate   34m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 35m   True    UpToDate   33m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                35m   True    UpToDate   33m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   35m   True    UpToDate   33m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              35m   True    UpToDate   33m

NAME                                                                     AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa   35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa          35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa                 35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa                   35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa                35m   True    UpToDate   35m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa                  35m   True    UpToDate   34m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa                  35m   True    UpToDate   34m

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-accesscontextmanager   35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-anthos                 35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudbilling           35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-cloudresourcemanager   35m   True    UpToDate   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-6475-serviceusage           35m   True    UpToDate   35m

Issues with

iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29m   False   DependencyNotFound   29m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29m   False   DependencyNotFound   29m

kubens config-control
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl get gcp | grep UpdateFailed
(not representative)

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions -n projects
Name:         security-log-bucket-writer-permissions
Namespace:    projects
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              config.k8s.io/owning-inventory: ace11e1affe3760bdf91752781e6fec950f9ba61-1701865095377759288
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206
              internal.kpt.dev/upstream-identifier: iam.cnrm.cloud.google.com|IAMPartialPolicy|projects|security-log-bucket-writer-permissions
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMPartialPolicy
Metadata:
  Creation Timestamp:  2023-12-06T12:22:34Z
  Generation:          1
  Resource Version:    25635
  UID:                 234ca697-ddd8-43a1-a3ed-e74f5a51d002
Spec:
  Bindings:
    Members:
      Member From:
        Log Sink Ref:
          Name:       org-log-sink-security-logging-project-oi1206
          Namespace:  logging
    Role:             roles/logging.bucketWriter
  Resource Ref:
    API Version:  resourcemanager.cnrm.cloud.google.com/v1beta1
    Kind:         Project
    Name:         logging-project-oi1206
    Namespace:    projects
Status:
  Conditions:
    Last Transition Time:  2023-12-06T12:22:35Z
    Message:               reference LoggingLogSink logging/org-log-sink-security-logging-project-oi1206 is not found
    Reason:                DependencyNotFound
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  DependencyNotFound  5m12s (x4 over 33m)  iampartialpolicy-controller  reference LoggingLogSink logging/org-log-sink-security-logging-project-oi1206 is not found
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ 

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-6475)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions -n projects
Name:         mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
Namespace:    projects
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              config.k8s.io/owning-inventory: ace11e1affe3760bdf91752781e6fec950f9ba61-1701865095377759288
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi1206
              internal.kpt.dev/upstream-identifier:
                iam.cnrm.cloud.google.com|IAMPartialPolicy|projects|mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMPartialPolicy
Metadata:
  Creation Timestamp:  2023-12-06T12:22:33Z
  Generation:          1
  Resource Version:    25600
  UID:                 bafe8b5a-8d98-466c-b76a-5e5f53ab509b
Spec:
  Bindings:
    Members:
      Member From:
        Log Sink Ref:
          Name:       mgmt-project-cluster-platform-and-component-log-sink
          Namespace:  logging
    Role:             roles/logging.bucketWriter
  Resource Ref:
    API Version:  resourcemanager.cnrm.cloud.google.com/v1beta1
    Kind:         Project
    Name:         logging-project-oi1206
    Namespace:    projects
Status:
  Conditions:
    Last Transition Time:  2023-12-06T12:22:34Z
    Message:               reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
    Reason:                DependencyNotFound
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  DependencyNotFound  103s (x5 over 37m)  iampartialpolicy-controller  reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found

Triage up the chain

Warning  DependencyNotFound  5m12s (x4 over 33m)  iampartialpolicy-controller  reference LoggingLogSink logging/org-log-sink-security-logging-project-oi1206 is not found

  Warning  DependencyNotFound  103s (x5 over 37m)  iampartialpolicy-controller  reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found

running main 20231206:0600 to verify no issue with the following https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/services-infrastructure/folder-sink.yaml#L23 platform-and-component-services-infra-log-sink from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/lz-folder/audits/logging-project/project-iam.yaml#L105 mgmt-project-cluster-platform-and-component-log-sink

fmichaelobrien commented 7 months ago

Summary:for shared billing accounts the projects-sa KCC focused SA must be associated as a BAU for direct accounts we are okhttps://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#5-perform-the-post-deployment-steps https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L332

without this the projects don't get created for shared billing users - and the log syncs as expected won't render until the project is up direct billing associates the SA with the BAU ok

actually their issue is clearer now - will add to the docs - need to wait for projects-sa
process....
kpt apply
wait 15 min or until projects-sa is up via krm - via wait(15 min) in the script
passed themcount=$(kubectl get gcp -n projects | grep UpdateFailed | wc -l)
wait until = 0 then proceed
associate projects-sa with kcc project for BAU (or they received an error that projects-sa does not yet exit)kpt re- apply
they will adjust their CD so earlier this week does not apply
obriensystems commented 5 months ago

retesting on mi*obr.ind

apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
data: 
  org-id: "459065442144"
  lz-folder-id: "388627537443"
  billing-id: "014479-806359-2F5F85"
  management-project-id: "kcc-oi-7970"
  management-project-number: "729005816584"
  management-namespace: config-control
  allowed-trusted-image-projects: |
    - "projects/cos-cloud"
  allowed-contact-domains: |
    - "@obrien.industries"
  allowed-policy-domain-members: |
    - "C03kdhrkc"
  allowed-vpc-peering: |
    - "under:organizations/459065442144"
  logging-project-id: logging-project-oi0130
  security-log-bucket: security-log-bucket-oi0130
  platform-and-component-log-bucket: platform-and-component-log-bucket-oi0130
  retention-locking-policy: "false"
  retention-in-days: "1"
  dns-project-id: dns-project-oi0130
  dns-name: "obrien.industries."

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@0.7.1
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.7.1 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live init core-landing-zone --namespace config-control 
initializing "resourcegroup.yaml" data (namespace: config-control)...success

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt fn render core-landing-zone --truncate-output=false
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
  Results:
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.projectRef.name: set field value to "logging-project-oi0130"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.name: set field value to "platform-and-component-log-bucket-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.projectRef.name: set field value to "logging-project-oi0130"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "logging-project-oi0130"
    [info] metadata.name: set field value to "kcc-oi-7970"
    [info] spec.metricsScope: set field value to "location/global/metricsScopes/logging-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.resourceRef.name: set field value to "logging-project-oi0130"
    [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "org-log-sink-security-logging-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.resourceRef.name: set field value to "logging-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.resourceRef.name: set field value to "logging-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.resourceRef.name: set field value to "logging-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.resourceRef.name: set field value to "logging-project-oi0130"
    [info] metadata.name: set field value to "logging-project-oi0130-data-access-sink"
    [info] spec.projectRef.name: set field value to "logging-project-oi0130"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi0130/locations/northamerica-northeast1/buckets/security-log-bucket"
    [info] metadata.name: set field value to "logging-project-oi0130"
    [info] spec.name: set field value to "logging-project-oi0130"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] metadata.name: set field value to "logging-project-oi0130-logging"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.projectRef.external: set field value to "logging-project-oi0130"
    [info] metadata.name: set field value to "logging-project-oi0130-monitoring"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi0130"
    [info] spec.projectRef.external: set field value to "logging-project-oi0130"
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi0130"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi0130/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi0130"
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.name: set field value to "dns-project-oi0130-standard-core-public-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi0130"
    [info] spec.dnsName: set field value to "obrien.industries."
    [info] metadata.name: set field value to "dns-project-oi0130"
    [info] spec.name: set field value to "dns-project-oi0130"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] metadata.name: set field value to "dns-project-oi0130-dns"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi0130"
    [info] spec.projectRef.external: set field value to "dns-project-oi0130"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi0130"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi0130/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi0130"
    [info] spec.folderRef.external: set field value to "388627537443"
    [info] metadata.name: set field value to "compute-disable-serial-port-logging-except-kcc-oi-7970"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.name: set field value to "compute-require-shielded-vm-except-kcc-oi-7970"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.name: set field value to "compute-restrict-cloud-nat-usage-except-kcc-oi-7970"
    [info] spec.listPolicy.allow.values[0]: set field value to "under:projects/kcc-oi-7970"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi0130"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi0130/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi0130"
    [info] metadata.name: set field value to "kcc-oi-7970-cloudbilling"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.name: set field value to "kcc-oi-7970-cloudresourcemanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.name: set field value to "kcc-oi-7970-serviceusage"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.name: set field value to "kcc-oi-7970-accesscontextmanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.name: set field value to "kcc-oi-7970-anthos"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.projectRef.external: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "kcc-oi-7970"
    [info] spec.member: set field value to "serviceAccount:config-mgmt-mon-default-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[config-management-monitoring/default]"
    [info] spec.googleServiceAccount: set field value to "config-mgmt-mon-default-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "kcc-oi-7970"
    [info] spec.member: set field value to "serviceAccount:gatekeeper-admin-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[gatekeeper-system/gatekeeper-admin]"
    [info] spec.googleServiceAccount: set field value to "gatekeeper-admin-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:hierarchy-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]"
    [info] spec.googleServiceAccount: set field value to "hierarchy-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-monitoring-admin-kcc-oi-7970-permissions"
    [info] metadata.namespace: set field value to "config-control"
    [info] spec.resourceRef.external: set field value to "kcc-oi-7970"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-monitoring-admin-logging-project-oi0130-permissions"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "logging-project-oi0130"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.name: set field value to "logging-sa-storageadmin-logging-project-oi0130-permissions"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "logging-project-oi0130"
    [info] spec.resourceRef.name: set field value to "logging-project-oi0130"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]"
    [info] spec.googleServiceAccount: set field value to "logging-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:service-729005816584@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "kcc-oi-7970"
    [info] spec.member: set field value to "serviceAccount:service-729005816584@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "kcc-oi-7970"
    [info] spec.member: set field value to "serviceAccount:service-729005816584@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]"
    [info] spec.googleServiceAccount: set field value to "networking-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:policies-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]"
    [info] spec.googleServiceAccount: set field value to "policies-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "388627537443"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-7970"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-7970.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]"
    [info] spec.googleServiceAccount: set field value to "projects-sa@kcc-oi-7970.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"under:organizations/459065442144\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/cos-cloud\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"@obrien.industries\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"C03kdhrkc\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] metadata.name: set field value to "org-log-sink-security-logging-project-oi0130"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi0130/locations/northamerica-northeast1/buckets/security-log-bucket"
    [info] metadata.name: set field value to "org-log-sink-data-access-logging-project-oi0130"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi0130/locations/northamerica-northeast1/buckets/security-log-bucket"

Successfully executed 1 function(s) in 1 package(s).
see
    external: kcc-oi-7970 # kpt-set: ${management-project-id}

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$  kpt live apply core-landing-zone --reconcile-timeout=15m --output=table

browser crashed halfway 10 min - after dual project creation

NAMESPACE RESOURCE ACTION STATUS RECONCILED CONDITIONS AGE MESSAGE
Namespace/hierarchy Successful Current 17m Resource is current
Namespace/logging Successful Current 17m Resource is current
Namespace/networking Successful Current 17m Resource is current
Namespace/policies Successful Current 17m Resource is current
Namespace/projects Successful Current 17m Resource is current
config-con IAMCustomRole/gke-firewall-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier2-dnsrecord-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier2-vpcpeering-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier3-dnsrecord-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier3-firewallrule-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier3-subnetwork-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier3-vpcsc-admin Successful Current Ready 17m Resource is Current
config-con IAMCustomRole/tier4-secretmanager-admin Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/config-mgmt-mon-default Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/gatekeeper-admin-sa-wor Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/hierarchy-sa-workload-i Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/logging-sa-workload-ide Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/networking-sa-workload- Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/policies-sa-workload-id Successful Current Ready 17m Resource is Current
config-con IAMPartialPolicy/projects-sa-workload-id Successful Current Ready 17m Resource is Current
config-con IAMPolicyMember/config-control-sa-manage Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/config-control-sa-manage Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/config-control-sa-orgrol Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/config-mgmt-mon-default- Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/gatekeeper-admin-sa-metr Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/hierarchy-sa-folderadmin Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/logging-sa-logadmin-perm Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/logging-sa-monitoring-ad Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/networking-sa-dns-permis Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/networking-sa-networkadm Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/networking-sa-security-p Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/networking-sa-service-co Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/networking-sa-servicedir Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/networking-sa-xpnadmin-p Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/policies-sa-orgpolicyadm Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/projects-sa-billinguser- Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/projects-sa-projectcreat Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/projects-sa-projectdelet Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/projects-sa-projectiamad Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/projects-sa-projectmover Successful Current Ready 16m Resource is Current
config-con IAMPolicyMember/projects-sa-serviceusage Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/config-mgmt-mon-defaul Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/gatekeeper-admin-sa Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/hierarchy-sa Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/logging-sa Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/networking-sa Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/policies-sa Successful Current Ready 16m Resource is Current
config-con IAMServiceAccount/projects-sa Successful Current Ready 16m Resource is Current
config-con Service/kcc-oi-7970-accesscontextmanager Successful Current Ready 16m Resource is Current
config-con Service/kcc-oi-7970-anthos Successful Current Ready 16m Resource is Current
config-con Service/kcc-oi-7970-cloudbilling Successful Current Ready 16m Resource is Current
config-con Service/kcc-oi-7970-cloudresourcemanager Successful Current Ready 16m Resource is Current
config-con Service/kcc-oi-7970-serviceusage Successful Current Ready 16m Resource is Current
config-man ConfigConnectorContext/configconnectorco Successful Current 17m status.healthy is true
gatekeeper ConfigConnectorContext/configconnectorco Successful Current 17m status.healthy is true
hierarchy ConfigConnectorContext/configconnectorco Successful Current 15m status.healthy is true
hierarchy RoleBinding/allow-folders-resource-refer Successful Current 15m Resource is current
hierarchy RoleBinding/allow-hierarchy-resource-ref Successful Current 15m Resource is current
hierarchy RoleBinding/allow-hierarchy-resource-ref Successful Current 15m Resource is current
hierarchy RoleBinding/allow-hierarchy-resource-ref Successful Current 15m Resource is current
hierarchy Folder/audits Successful Current Ready 15m Resource is Current
hierarchy Folder/clients Successful Current Ready 15m Resource is Current
hierarchy Folder/services Successful Current Ready 15m Resource is Current
hierarchy Folder/services-infrastructure Successful Current Ready 15m Resource is Current
logging ConfigConnectorContext/configconnectorco Successful Current 15m status.healthy is true
logging LoggingLogBucket/platform-and-component- Successful Current Ready 12m Resource is Current
logging LoggingLogBucket/security-log-bucket Successful Current Ready 12m Resource is Current
logging LoggingLogSink/logging-project-oi0130-da Pending Unknown - -
logging LoggingLogSink/mgmt-project-cluster-plat Pending Unknown - -
logging LoggingLogSink/org-log-sink-data-access- Pending Unknown - -
logging LoggingLogSink/org-log-sink-security-log Pending Unknown - -
logging LoggingLogSink/platform-and-component-se Pending Unknown - -
logging LoggingLogSink/platform-and-component-se Pending Unknown - -
logging MonitoringMonitoredProject/kcc-oi-7970 Successful Current Ready 15m Resource is Current
logging RoleBinding/allow-logging-resource-refer Successful Current 15m Resource is current
logging StorageBucket/security-incident-log-buck Successful Failed Ready 12m Update call failed: error fetching live networking ConfigConnectorContext/configconnectorco Successful Current 15m status.healthy is true
networking DNSManagedZone/dns-project-oi0130-standa Pending Unknown - -
policies ConfigConnectorContext/configconnectorco Successful Current 15m status.healthy is true
policies ResourceManagerPolicy/compute-disable-gu Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-disable-ne Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-disable-se Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-disable-se Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-disable-se Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-disable-vp Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-require-os Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-require-sh Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-require-sh Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-restrict-c Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-restrict-c Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-restrict-l Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-restrict-s Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-restrict-v Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-restrict-v Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-skip-defau Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-trusted-im Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-vm-can-ip- Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/compute-vm-externa Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/essentialcontacts- Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/gcp-restrict-resou Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/iam-allowed-policy Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/iam-automatic-iam- Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/iam-disable-audit- Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/iam-disable-servic Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/iam-disable-servic Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/sql-restrict-publi Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/storage-public-acc Successful Current Ready 15m Resource is Current
policies ResourceManagerPolicy/storage-uniform-bu Successful Current Ready 15m Resource is Current
projects ConfigConnectorContext/configconnectorco Successful Current 15m status.healthy is true
projects IAMAuditConfig/logging-project-data-acce Successful Current Ready 12m Resource is Current
projects IAMPartialPolicy/mgmt-project-cluster-pl Successful InProgress Ready 12m reference LoggingLogSink logging/mgmt-pr projects IAMPartialPolicy/platform-and-component- Successful InProgress Ready 12m reference LoggingLogSink logging/platfor projects IAMPartialPolicy/platform-and-component- Successful InProgress Ready 12m reference LoggingLogSink logging/platfor projects IAMPartialPolicy/security-log-bucket-wri Successful InProgress Ready 12m reference LoggingLogSink logging/org-log projects IAMPolicyMember/logging-sa-monitoring-ad Successful Current Ready 15m Resource is Current
projects IAMPolicyMember/logging-sa-storageadmin- Successful Current Ready 15m Resource is Current
projects RoleBinding/allow-projects-resource-refe Successful Current 15m Resource is current
projects RoleBinding/allow-projects-resource-refe Successful Current 15m Resource is current
projects RoleBinding/allow-projects-resource-refe Successful Current 15m Resource is current
projects Project/dns-project-oi0130 Successful Current Ready 12m Resource is Current
projects Project/logging-project-oi0130 Successful Current Ready 15m Resource is Current
projects Service/dns-project-oi0130-dns Pending Unknown - -
projects Service/logging-project-oi0130-logging Successful Current Ready 12m Resource is Current
projects Service/logging-project-oi0130-monitorin Successful Current Ready 12m Resource is Current

13 over 12 michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone | grep not inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/org-log-sink-security-logging-project-oi0130 is not found inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-log-sink is not found inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-infra-log-sink is not found inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-oi0130-data-access-sink is NotFound: Resource not found inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is NotFound: Resource not found inventory-36746767/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-oi0130-standard-core-public-dns is NotFound: Resource not found inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-oi0130-dns is NotFound: Resource not found inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is NotFound: Resource not found inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is NotFound: Resource not found inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-oi0130 is NotFound: Resource not found inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-oi0130 is NotFound: Resource not found

obriensystems commented 5 months ago

recheck cluster - time heals - just needed an extra hour

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone 
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-36746767/logginglogbucket.logging.cnrm.cloud.google.com/logging/security-log-bucket is Current: Resource is Current
inventory-36746767/logginglogbucket.logging.cnrm.cloud.google.com/logging/platform-and-component-log-bucket-oi0130 is Current: Resource is Current
inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden
inventory-36746767/monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/logging/kcc-oi-7970 is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is Current: Resource is Current
inventory-36746767/iamauditconfig.iam.cnrm.cloud.google.com/projects/logging-project-data-access-log-config is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-oi0130-data-access-sink is Current: Resource is Current
inventory-36746767/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-oi0130 is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-oi0130-logging is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/logging-project-oi0130-monitoring is Current: Resource is Current
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is Current: Resource is Current
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-36746767/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-oi0130-standard-core-public-dns is Current: Resource is Current
inventory-36746767/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-oi0130 is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-oi0130-dns is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is Current: Resource is Current
inventory-36746767/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging-except-kcc-oi-7970 is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-kcc-oi-7970 is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-kcc-oi-7970 is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-cloudbilling is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-cloudresourcemanager is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-serviceusage is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-accesscontextmanager is Current: Resource is Current
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-oi-7970-anthos is Current: Resource is Current
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-metric-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/config-mgmt-mon-default-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/config-management-monitoring/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//hierarchy is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-monitoring-admin-kcc-oi-7970-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-monitoring-admin-logging-project-oi0130-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/projects/logging-sa-storageadmin-logging-project-oi0130-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//logging is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//networking is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//policies is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-36746767/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-36746767/namespace//projects is Current: Resource is current
inventory-36746767/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-36746767/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-36746767/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-subnetwork-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-36746767/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier4-secretmanager-admin is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-logging is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpn-peer-ips is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-automatic-iam-grants-for-default-service-accounts is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-audit-logging-exemption is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-upload is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-36746767/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-oi0130 is Current: Resource is Current
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-oi0130 is Current: Resource is Current

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone | grep not
inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get namespaces
NAME                              STATUS   AGE
cnrm-system                       Active   17h
config-control                    Active   17h
config-management-monitoring      Active   17h
config-management-system          Active   17h
configconnector-operator-system   Active   17h
configsync-healthcheck-system     Active   17h
default                           Active   18h
gatekeeper-system                 Active   17h
gke-gmp-system                    Active   17h
gke-managed-filestorecsi          Active   17h
gmp-public                        Active   17h
hierarchy                         Active   17h
krmapihosting-monitoring          Active   17h
krmapihosting-system              Active   17h
kube-node-lease                   Active   18h
kube-public                       Active   18h
kube-system                       Active   18h
logging                           Active   17h
networking                        Active   17h
policies                          Active   17h
projects                          Active   17h
resource-group-system             Active   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   17h   True    UpToDate   17h

NAME                                                                                                                   AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   17h   True    UpToDate   16h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         17h   True    UpToDate   16h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               17h   True    UpToDate   16h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      17h   True    UpToDate   16h

NAME                                                                                                       AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi0130-permissions   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi0130-permissions       17h   True    UpToDate   17h

NAME                                                                   AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi0130       17h   True    UpToDate   17h
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi0130   17h   True    UpToDate   17h

NAME                                                                           AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/dns-project-oi0130-dns              16h   True    UpToDate   16h
service.serviceusage.cnrm.cloud.google.com/logging-project-oi0130-logging      17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/logging-project-oi0130-monitoring   17h   True    UpToDate   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n networking
NAME                                                                                   AGE   READY   STATUS     STATUS AGE
dnsmanagedzone.dns.cnrm.cloud.google.com/dns-project-oi0130-standard-core-public-dns   16h   True    UpToDate   16h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n logging
NAME                                                                                      AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi0130   17h   True    UpToDate   17h
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                        17h   True    UpToDate   17h

NAME                                                                                                AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/logging-project-oi0130-data-access-sink                16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-sink   16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-data-access-logging-project-oi0130        16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-security-logging-project-oi0130           16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-infra-log-sink         16h   True    UpToDate   16h
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-log-sink               16h   True    UpToDate   16h

NAME                                                                      AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-7970   17h   True    UpToDate   17h

NAME                                                                       AGE   READY   STATUS         STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket   17h   False   UpdateFailed   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    17h   True    UpToDate   17h
folder.resourcemanager.cnrm.cloud.google.com/clients                   17h   True    UpToDate   17h
folder.resourcemanager.cnrm.cloud.google.com/services                  17h   True    UpToDate   17h
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   17h   True    UpToDate   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n policies
NAME                                                                                                                 AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access                   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization                    17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access                       17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging                      17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-7970   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6                        17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login                                 17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm                              17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-7970           17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage                         17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-7970      17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types        17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal                 17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering                             17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpn-peer-ips                            17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation                    17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects                           17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward                                17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access                            17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains                17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations                          17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains                        17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-automatic-iam-grants-for-default-service-accounts    17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-audit-logging-exemption                      17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation                 17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-upload                   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip                                   17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention                         17h   True    UpToDate   17h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access                      17h   True    UpToDate   17h
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kubectl get gcp -n config-control
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           17h   True    UpToDate   17h
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   17h   True    UpToDate   17h

NAME                                                                                              AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding   17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding          17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding                 17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding                   17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding                17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding                  17h   True    UpToDate   17h
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding                  17h   True    UpToDate   17h

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions                   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-7970-permissions                    17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   17h   True    UpToDate   17h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              17h   True    UpToDate   17h

NAME                                                                     AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa   17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa          17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa                 17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa                   17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa                17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa                  17h   True    UpToDate   17h
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa                  17h   True    UpToDate   17h

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-accesscontextmanager   17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-anthos                 17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-cloudbilling           17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-cloudresourcemanager   17h   True    UpToDate   17h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-serviceusage           17h   True    UpToDate   17h

looking into the single failure
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live status core-landing-zone | grep not
inventory-36746767/storagebucket.storage.cnrm.cloud.google.com/logging/security-incident-log-bucket is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Storage Bucket "security-incident-log-bucket": googleapi: Error 403: logging-sa@kcc-oi-7970.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden

rerunning on other cloud-setup
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@0.7.1
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.7.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.7.1 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ cp core-landing-zone/org/
custom-roles/  org-policies/  org-sink.yaml  
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ cp -R core-landing-zone/org/org-policies/ ../core-landing-zone/org/
michael@cloudshell:~/kcc-cso/kpt/_temp (kcc-cso-4380)$ cd ../
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render core-landing-zone --truncate-output=false
obriensystems commented 5 months ago

needed org-policies folder and a 1 hour wait state