GoogleCloudPlatform / pubsec-declarative-toolkit

The GCP PubSec Declarative Toolkit is a collection of declarative solutions to help you on your Journey to Google Cloud. Solutions are designed using Config Connector and deployed using Config Controller.

Apache License 2.0

30 stars 27 forks source link

Example 258 fortigate perimeter package deploy procedure/verify for core lz unmanaged client #446

Open fmichaelobrien opened 11 months ago

fmichaelobrien commented 11 months ago

updates

review alternative serverless approach using https://cloud.google.com/blog/products/identity-security/introducing-google-cloud-firewall-plus-with-intrusion-prevention
20240125: move setup.sh to merge in phases in #766
20240220: investigate policy based routing for the hub-env package VPC - https://cloud.google.com/vpc/docs/policy-based-routes
- refer to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494 dev = root landing systems test = nuage finops dev.. nua.cl.o
training https://www.cloudskillsboost.google/focuses/77469?catalog_rank=%7B%22rank%22%3A1%2C%22num_filters%22%3A0%2C%22has_search%22%3Atrue%7D&parent=catalog&search_id=29853894

FinOps: PAYG + GKE + GCE costs will be $80/day above the normal $10/day for the GKE cluster alone.

The client requires deployment of the #258 perimeter on top of the core lz with additional DNS zones TBD

Document and reuse on top of #420 and and #421 gcloud deployment testing later 2022 - #158 See pre-kcc deployment run in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/158

gcloud reference install: https://github.com/fortinet/fortigate-tutorial-gcp/issues/1

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps

graph LR;
    style LZV2 fill:#44f,stroke:#f66,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
    %% mapped and documented
    project/hub-env-->core-landing-zone;
    client-setup;
    client-setup-->dns-project;
    client-setup-->kcc-management-project;
    client-landing-zone-->client-setup;
    client-project-setup-->client-landing-zone;
    client-project-setup-->client-management-project;
    gatekeeper-policies;

    kcc-management-project;
    core-landing-zone-->kcc-management-project;
    dns-project-->core-landing-zone;
    logging-project-->core-landing-zone;
    client-management-project-->client-setup;
    host-project-->client-landing-zone;

mermaid - diagrams as code See

20231027 clean org demo run of core-landing-zone and hub-env https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/611
78
168
166
177
207
573

todo:
add client-landing-zone on top of existing core-landing-zone before hub-env

Package Inventory

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Architecture

Notes:

.sh linter failures https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/728

fmichaelobrien commented 11 months ago

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env/fortigate https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone

no interconnects

obriensystems commented 11 months ago

org states

kcc.landing.systems

3 month old main core-landing-zone
3 month old hub-env partially up with a fixes
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/574
573
obrien.industries
dev box only for automation creation
1 week old - was master - now 0.3.2 core-landing-zone - more stable
8+ lz reinstalls - system unclean for hub-env

landing systems

target test box only for automation of lcls
full automation via sh script only
derived setters.yaml
20231022 0.3.2 core-landing-zone
will manually run hub-env

new org

deploy clz + hub-env together as changes come in from ls above
TODO
add an onboarding readme for the hub-env like
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-admin.md

edit https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/setters.yaml via https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/README.md

Known Issues Workarounds

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigates.md#known-issues
turn down security on the management VPC - turn back on public access for googleapis.com public access without putting in a private service connect endpoint like
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/468

obriensystems commented 11 months ago

know oci and gitops are the core - and I agree having the code in github/ado/gitlab/csr is preferred - but some clients have requested the easier kpt option and it should be there as a base deployment option - since mid 2022

To be fair the base case deployment option is actually pure kubernetes krm yaml like in https://cloud.google.com/config-connector/docs/how-to/getting-started

see original gitops docs in https://cloud.google.com/anthos-config-management/docs/concepts/config-controller-overview and https://cloud.google.com/anthos-config-management/docs/how-to/unstructured-repo see original kpt docs in https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt

20230814: revisit kls

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ export PROJECT_ID=kcc-kls-cluster3
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ ls
core-landing-zone  setters.yaml
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ rm -rf core-landing-zone/
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@main
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 + 52f93a3...ea2e57f main       -> origin/main  (forced update)
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ cp setters.yaml core-landing-zone/

re-add kpt documentation at the end of section 2 see #409

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#2-create-your-landing-zone needs https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/solutions/core-landing-zone/0.3.0/docs/landing-zone-v2#kpt

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$   kpt live init core-landing-zone --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render core-landing-zone
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[FAIL] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.1s
  Results:
    [error]: failed to apply setters: values for setters [${platform-and-component-log-bucket}] must be provided
  Stderr:
    "values for setters [${platform-and-component-log-bucket}] must be providedvalues for setters [${platform-and-component-log-bucket}] must be provided"
  Exit code: 1

  fix: did not have the latest version of setters.yaml - updated

  mirroring changes to my local repo from the core-landing-zone kpt folder download

root_@cloudshell:~/kcc-kls/lz-20230803-gh/pubsec-declarative-toolkit (kcc-kls-cluster3)$ git diff
diff --git a/solutions/core-landing-zone/setters.yaml b/solutions/core-landing-zone/setters.yaml
index f3168d3..ca53ae4 100644
--- a/solutions/core-landing-zone/setters.yaml
+++ b/solutions/core-landing-zone/setters.yaml
@@ -14,10 +14,11 @@
 #########
 apiVersion: v1
 kind: ConfigMap
-metadata:
+metadata: # kpt-merge: /setters
   name: setters
   annotations:
     config.kubernetes.io/local-config: "true"
+    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
 data:
   ##########################
   # Instructions
@@ -38,9 +39,9 @@ data:
   # General Settings Values
   ##########################
   #
-  org-id: "0000000000"
-  lz-folder-id: '0000000000'
-  billing-id: "AAAAAA-BBBBBB-CCCCCC"
+  org-id: "15....993"
+  lz-folder-id: '444....332'
+  billing-id: "01....833"
   #
   ##########################
   # Management Project
@@ -48,8 +49,8 @@ data:
   #
   # This is the project where the config controller instance is running
   # Values can be viewed in the Project Dashboard
-  management-project-id: management-project-12345
-  management-project-number: "0000000000"
+  management-project-id: kcc-kls-cluster3
+  management-project-number: "53....547"
   management-namespace: config-control
   #
   ##########################
@@ -68,14 +69,14 @@ data:
   # org/org-policies/essentialcontacts-allowed-contact-domains.yaml
   # this setting MUST be changed
   allowed-contact-domains: |
-    - "@example.com"
+    - "@kcc.landing.systems"
   #
   # a list of directory customer IDs from which users can be added to IAM policies, see YAML file for more info:
   # org/org-policies/iam-allowed-policy-member-domains.yaml
   # this setting MUST be changed to include the GCP org's directory ID and any other directory containing users that will need IAM roles assigned
   # run 'gcloud organizations list' as described in https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains#retrieving_customer_id
   allowed-policy-domain-members: |
-    - "DIRECTORY_CUSTOMER_ID"
+    - "C0....m1"
   #
   # a list of allowed projects, folders, networks for VPC peering, see YAML file for more info:
   # org/org-policies/compute-restrict-vpc-peering.yaml
@@ -87,13 +88,13 @@ data:
   # Logging
   ##########################
   #
-  logging-project-id: logging-project-12345
+  logging-project-id: logging-project-kls
   #
   # Log Buckets
   # Security Logs Bucket
-  security-log-bucket: security-log-bucket-12345
+  security-log-bucket: security-log-bucket-kls
   # Platform and Component Log Bucket
-  platform-and-component-log-bucket: platform-and-component-log-bucket-12345
+  platform-and-component-log-bucket: platform-and-component-log-bucket-kls
   #
   # Retention settings
   # Set the number of days to retain logs in Cloud Logging buckets
@@ -110,8 +111,9 @@ data:
   # DNS
   ##########################
   #
-  dns-project-id: dns-project-12345
-  dns-name: "example.com."
+  dns-project-id: dns-project-kls
+  # the appended . is required by google cloud domain zones
+  dns-name: "kcc.landing.systems."

kpt rendering ok

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render core-landing-zone
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 600ms
  Results:
    [info] spec.folderRef.external: set field value to "444332200332"
    [info] metadata.name: set field value to "security-log-bucket-kls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-kls"
    [info] spec.projectRef.name: set field value to "logging-project-kls"
    ...(213 line(s) truncated, use '--truncate-output=false' to disable)

Successfully executed 1 function(s) in 1 package(s).

kpt live apply (20230414:1552)

kpt live apply core-landing-zone --reconcile-timeout=2m --output=table

coming up

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    2m      Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    2m      Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Failed                  Ready                                     2m      Update call failed: error setting policy
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/hierarchy-sa            Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/logging-sa              Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/networking-sa           Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/policies-sa             Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  IAMServiceAccount/projects-sa             Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-accesscontextma  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudbilling     Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudresourcema  Successful    Current                 Ready                                     2m      Resource is Current                     
config-con  Service/kcc-kls-cluster3-serviceusage     Successful    Current                 Ready                                     2m      Resource is Current                     
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    37s     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    37s     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    37s     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    36s     Resource is current                     
hierarchy   Folder/audits                             Successful    Current                 Ready                                     34s     Resource is Current                     
hierarchy   Folder/clients                            Successful    Current                 Ready                                     33s     Resource is Current                     
hierarchy   Folder/services                           Successful    Current                 Ready                                     33s     Resource is Current                     
hierarchy   Folder/services-infrastructure            Successful    Current                 Ready                                     33s     Resource is Current                     
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogBucket/security-log-bucket-kls  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/logging-project-kls-secur  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-disa  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-plat  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    36s     Resource is current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
networking  DNSManagedZone/dns-project-kls-standard-  Skipped       Unknown                 -                                         -                                               
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-os  Successful    Current                 Ready                                     32s     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Current                 Ready                                     31s     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Failed                  Ready                                     30s     Update call failed: error applying desir
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     30s     Resource is Current                     
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Current                 Ready                                     29s     Resource is Current                     
policies    ResourceManagerPolicy/storage-public-acc  Successful    Current                 Ready                                     28s     Resource is Current                     
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Current                 Ready                                     28s     Resource is Current                     
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    35s     status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/security-log-bucket-wri  Skipped       Unknown                 -                                         -                                               
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    36s     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    36s     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    36s     Resource is current                     
projects    Project/dns-project-kls                   Successful    Failed                  Ready                                     2s      Update call failed: error fetching live 
projects    Project/logging-project-kls               Successful    Failed                  Ready                                     33s     Update call failed: error fetching live 
projects    Service/dns-project-kls-dns               Skipped       Unknown                 -                                         -                                               

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                       READY   STATUS    RESTARTS       AGE
cnrm-system                       cnrm-controller-manager-3fo6phebqgg23knqq5qq-0             2/2     Running   0              4m2s
cnrm-system                       cnrm-controller-manager-7c4rehlik7xgxc2utq6a-0             2/2     Running   0              4m2s
cnrm-system                       cnrm-controller-manager-fiqj4dqbgpwy6mlvh25q-0             2/2     Running   0              4m
cnrm-system                       cnrm-controller-manager-ghhiigeeussitzq7mfza-0             2/2     Running   0              4m
cnrm-system                       cnrm-controller-manager-gnunqke5gjhr55wngr7q-0             2/2     Running   0              4m1s
cnrm-system                       cnrm-controller-manager-sgfj3cxgisp6jdsfy7qq-0             2/2     Running   0              5d3h
cnrm-system                       cnrm-controller-manager-swyfekd4gcdftjnvc2qa-0             2/2     Running   0              5m29s
cnrm-system                       cnrm-deletiondefender-0                                    1/1     Running   0              5d3h
cnrm-system                       cnrm-resource-stats-recorder-88bfdfd56-kqdq2               2/2     Running   0              5d3h
cnrm-system                       cnrm-unmanaged-detector-0                                  1/1     Running   0              5d3h
cnrm-system                       cnrm-webhook-manager-54c8477885-cr54f                      1/1     Running   0              5d3h
cnrm-system                       cnrm-webhook-manager-54c8477885-plgkd                      1/1     Running   0              4m36s
cnrm-system                       cnrm-webhook-manager-54c8477885-ssldj                      1/1     Running   0              5d3h
config-management-monitoring      otel-collector-865b4f4968-l89bt                            1/1     Running   0              5d3h
config-management-system          config-management-operator-5db59f7f8f-5fb4p                1/1     Running   0              5d3h
config-management-system          reconciler-manager-5cddc57f5-bxc86                         2/2     Running   0              5d3h
configconnector-operator-system   configconnector-operator-0                                 1/1     Running   0              5d3h
gatekeeper-system                 gatekeeper-audit-6d686f5467-zlwzr                          1/1     Running   0              5d3h
gatekeeper-system                 gatekeeper-controller-manager-6b47854cf5-nsmzs             1/1     Running   0              5d3h
gke-gmp-system                    alertmanager-0                                             2/2     Running   2 (11d ago)    11d
gke-gmp-system                    collector-bb4st                                            2/2     Running   2 (11d ago)    11d
gke-gmp-system                    collector-h4j24                                            2/2     Running   1 (11d ago)    11d
gke-gmp-system                    collector-szhxn                                            2/2     Running   2 (11d ago)    11d
gke-gmp-system                    gmp-operator-7645bc584f-5d8gf                              1/1     Running   0              30h
gke-gmp-system                    rule-evaluator-767c5ccc99-7mbnt                            2/2     Running   2 (11d ago)    11d
krmapihosting-monitoring          krmapihosting-metrics-agent-55glj                          1/1     Running   0              11d
krmapihosting-monitoring          krmapihosting-metrics-agent-9nlw9                          1/1     Running   0              11d
krmapihosting-monitoring          krmapihosting-metrics-agent-d8xm9                          1/1     Running   0              11d
krmapihosting-system              bootstrap-5d5578f758-sh76w                                 1/1     Running   0              5d3h
kube-system                       anetd-cg6g9                                                1/1     Running   0              11d
kube-system                       anetd-f2gpt                                                1/1     Running   0              11d
kube-system                       anetd-r7gr2                                                1/1     Running   0              11d
kube-system                       antrea-controller-horizontal-autoscaler-7b69d9bfd7-rqq8r   1/1     Running   0              11d
kube-system                       egress-nat-controller-98648bc69-fm8nk                      1/1     Running   0              11d
kube-system                       event-exporter-gke-7bf6c99dcb-c5dd9                        2/2     Running   0              11d
kube-system                       filestore-node-4p9cx                                       3/3     Running   0              11d
kube-system                       filestore-node-5jlfv                                       3/3     Running   0              11d
kube-system                       filestore-node-74pm4                                       3/3     Running   1 (7d6h ago)   11d
kube-system                       fluentbit-gke-big-6hsk5                                    2/2     Running   0              11d
kube-system                       fluentbit-gke-big-sxkh2                                    2/2     Running   0              11d
kube-system                       fluentbit-gke-big-vm26j                                    2/2     Running   0              11d
kube-system                       gcsfusecsi-node-7k76l                                      2/2     Running   0              11d
kube-system                       gcsfusecsi-node-j8r4b                                      2/2     Running   0              11d
kube-system                       gcsfusecsi-node-sq62q                                      2/2     Running   0              11d
kube-system                       gke-metadata-server-btb9x                                  1/1     Running   0              30h
kube-system                       gke-metadata-server-l447p                                  1/1     Running   0              30h
kube-system                       gke-metadata-server-w7brs                                  1/1     Running   0              30h
kube-system                       gke-metrics-agent-9hvwg                                    2/2     Running   0              11d
kube-system                       gke-metrics-agent-j4xvr                                    2/2     Running   0              11d
kube-system                       gke-metrics-agent-spdl8                                    2/2     Running   0              11d
kube-system                       ip-masq-agent-cphwd                                        1/1     Running   0              11d
kube-system                       ip-masq-agent-n7nbw                                        1/1     Running   0              11d
kube-system                       ip-masq-agent-r8pvq                                        1/1     Running   0              11d
kube-system                       konnectivity-agent-5b687c8dcb-d64h7                        1/1     Running   0              5d3h
kube-system                       konnectivity-agent-5b687c8dcb-dkrth                        1/1     Running   0              11d
kube-system                       konnectivity-agent-5b687c8dcb-vgmkm                        1/1     Running   0              11d
kube-system                       konnectivity-agent-autoscaler-5d9dbcc6d8-2s5dp             1/1     Running   0              11d
kube-system                       kube-dns-865c4fb86d-k5b2c                                  4/4     Running   0              11d
kube-system                       kube-dns-865c4fb86d-skmk6                                  4/4     Running   0              11d
kube-system                       kube-dns-autoscaler-84b8db4dc7-h47j6                       1/1     Running   0              11d
kube-system                       l7-default-backend-58c4fb8884-7n45b                        1/1     Running   0              2d6h
kube-system                       metrics-server-v0.5.2-6bf74b5d5f-fknxl                     2/2     Running   0              11d
kube-system                       netd-dtqvj                                                 1/1     Running   0              11d
kube-system                       netd-l5wgc                                                 1/1     Running   0              11d
kube-system                       netd-nhgl9                                                 1/1     Running   0              11d
kube-system                       node-local-dns-5wzzk                                       1/1     Running   0              11d
kube-system                       node-local-dns-bxqzh                                       1/1     Running   0              11d
kube-system                       node-local-dns-fkfln                                       1/1     Running   0              11d
kube-system                       pdcsi-node-h8jzw                                           2/2     Running   0              9d
kube-system                       pdcsi-node-hl6m6                                           2/2     Running   0              9d
kube-system                       pdcsi-node-sxfns                                           2/2     Running   0              9d
resource-group-system             resource-group-controller-manager-5594cd7b8-l87bc          2/2     Running   0              5d3h

just 1 org policy has an issue missed a setters.yaml var under:organizations/ORGANIZATION_ID]


  allowed-vpc-peering: |
    - "under:organizations/15..."

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt live status core-landing-zone
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-kls is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-kls is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-mgt-project is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudbilling is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudresourcemanager is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-serviceusage is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-accesscontextmanager is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//hierarchy is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-bigqueryadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//logging is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//networking is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//policies is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//projects is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Failed: Update call failed: error applying desired state: summary: googleapi: Error 400: One or more values is invalid.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.BadRequest",
    "fieldViolations": [
      {
        "description": "Invalid value: [under:organizations/ORGANIZATION_ID]",
        "field": "policy.list_policy.allowed_values[0]"
      }
    ]
  }
]
, badRequest
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current

fixing

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    61m     Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    61m     Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/hierarchy-sa            Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/logging-sa              Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/networking-sa           Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/policies-sa             Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  IAMServiceAccount/projects-sa             Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-accesscontextma  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudbilling     Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-cloudresourcema  Successful    Current                 Ready                                     61m     Resource is Current                     
config-con  Service/kcc-kls-cluster3-serviceusage     Successful    Current                 Ready                                     61m     Resource is Current                     
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    61m     status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    60m     Resource is current                     
hierarchy   Folder/audits                             Successful    Current                 Ready                                     60m     Resource is Current                     
hierarchy   Folder/clients                            Successful    Current                 Ready                                     60m     Resource is Current                     
hierarchy   Folder/services                           Successful    Current                 Ready                                     60m     Resource is Current                     
hierarchy   Folder/services-infrastructure            Successful    Current                 Ready                                     60m     Resource is Current                     
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Successful    Current                 Ready                                     3m      Resource is Current                     
logging     LoggingLogBucket/security-log-bucket-kls  Successful    Current                 Ready                                     3m      Resource is Current                     
logging     LoggingLogSink/logging-project-kls-secur  Successful    Current                 Ready                                     32s     Resource is Current                     
logging     LoggingLogSink/mgmt-project-cluster-disa  Successful    Current                 Ready                                     3m      Resource is Current                     
logging     LoggingLogSink/mgmt-project-cluster-plat  Successful    Current                 Ready                                     31s     Resource is Current                     
logging     LoggingLogSink/platform-and-component-se  Successful    Current                 Ready                                     31s     Resource is Current                     
logging     LoggingLogSink/platform-and-component-se  Successful    Current                 Ready                                     31s     Resource is Current                     
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    60m     Resource is current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
networking  DNSManagedZone/dns-project-kls-standard-  Successful    InProgress              Ready                                     32s     Update in progress                      
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-os  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/storage-public-acc  Successful    Current                 Ready                                     60m     Resource is Current                     
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Current                 Ready                                     60m     Resource is Current                     
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    60m     status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/platform-and-component-  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/platform-and-component-  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    IAMPartialPolicy/security-log-bucket-wri  Successful    Current                 Ready                                     3m      Resource is Current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    60m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    60m     Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    60m     Resource is current                     
projects    Project/dns-project-kls                   Successful    Current                 Ready                                     59m     Resource is Current                     
projects    Project/logging-project-kls               Successful    Current                 Ready                                     60m     Resource is Current                     
projects    Service/dns-project-kls-dns               Successful    Current                 Ready                                     30s     Resource is Current                     

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt live status core-landing-zone
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/audits is Current: Resource is Current
inventory-85852139/logginglogbucket.logging.cnrm.cloud.google.com/logging/security-log-bucket-kls is Current: Resource is Current
inventory-85852139/logginglogbucket.logging.cnrm.cloud.google.com/logging/platform-and-component-log-bucket-kls is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is Current: Resource is Current
inventory-85852139/iamauditconfig.iam.cnrm.cloud.google.com/projects/logging-project-data-access-log-config is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/logging-project-kls is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/clients is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services is Current: Resource is Current
inventory-85852139/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-kls-standard-core-public-dns is Current: Resource is Current
inventory-85852139/project.resourcemanager.cnrm.cloud.google.com/projects/dns-project-kls is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-kls-dns is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is Current: Resource is Current
inventory-85852139/folder.resourcemanager.cnrm.cloud.google.com/hierarchy/services-infrastructure is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-mgt-project is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-disable-default-bucket is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudbilling is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-cloudresourcemanager is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-serviceusage is Current: Resource is Current
inventory-85852139/service.serviceusage.cnrm.cloud.google.com/config-control/kcc-kls-cluster3-accesscontextmanager is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-metric-writer-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/gatekeeper-admin-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/hierarchy-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-folderadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/hierarchy-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//hierarchy is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/hierarchy/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-projects is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-policies is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-hierarchy-resource-reference-from-config-control is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/hierarchy/allow-folders-resource-reference-to-logging is Current: Resource is current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/logging-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-logadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/logging-sa-bigqueryadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/logging-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//logging is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/logging/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/logging/allow-logging-resource-reference-from-projects is Current: Resource is current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-orgroleadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-editor-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/config-control-sa-management-project-serviceaccountadmin-permissions is Current: Resource is Current
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/networking-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-networkadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-security-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-dns-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-service-control-org-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-xpnadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-servicedirectoryeditor-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/networking-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//networking is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/networking/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/policies-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/policies-sa-orgpolicyadmin-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/policies-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//policies is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/policies/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/iamserviceaccount.iam.cnrm.cloud.google.com/config-control/projects-sa is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectiamadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectcreator-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectmover-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-projectdeleter-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-serviceusageadmin-permissions is Current: Resource is Current
inventory-85852139/iampolicymember.iam.cnrm.cloud.google.com/config-control/projects-sa-billinguser-permissions is Current: Resource is Current
inventory-85852139/iampartialpolicy.iam.cnrm.cloud.google.com/config-control/projects-sa-workload-identity-binding is Current: Resource is Current
inventory-85852139/namespace//projects is Current: Resource is current
inventory-85852139/configconnectorcontext.core.cnrm.cloud.google.com/projects/configconnectorcontext.core.cnrm.cloud.google.com is Current: status.healthy is true
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-logging is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-networking is Current: Resource is current
inventory-85852139/rolebinding.rbac.authorization.k8s.io/projects/allow-projects-resource-reference-from-policies is Current: Resource is current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/gke-firewall-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier2-vpcpeering-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-dnsrecord-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-firewallrule-admin is Current: Resource is Current
inventory-85852139/iamcustomrole.iam.cnrm.cloud.google.com/config-control/tier3-vpcsc-admin is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-guest-attribute-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-nested-virtualization is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-vpc-external-ipv6 is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-os-login is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-shared-vpc-lien-removal is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current
inventory-85852139/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-kls-security-sink is Current: Resource is Current
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$

deploying hub package

kubectl get gcp --all-namespaces
kubectl get gcp -n projects

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/project/hub-env@main
Package "hub-env":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
Adding package "solutions/project/hub-env".

Fetched 1 package(s).

modifying setters
 apiVersion: v1
 kind: ConfigMap
-metadata:
+metadata: # kpt-merge: /setters
   name: setters
   annotations:
     config.kubernetes.io/local-config: "true"
+    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
 data:
   # Billing Account ID to be associated with this project
-  project-billing-id: "AAAAAA-BBBBBB-CCCCCC"
+  project-billing-id: "01A4...699F"
   # GCP folder to use as parent to this project, lowercase K8S resource name
-  project-parent-folder: project-parent-folder
+  project-parent-folder: services-infrastructure
   # Naming Convention for project-id : <tenant-code><environment-code>m<data-classification>-<project-owner>-<user defined string>
-  # Max 30 characters
-  hub-project-id: xxdmu-admin1-projectname
+  # Max 30 characters - must be unique for name to match id
+  hub-project-id: dmu-admin1-hub-kls
   # Identity that should be allowed to access the management VM using IAP TCP forwarding
   # https://cloud.google.com/iap/docs/using-tcp-forwarding
-  hub-admin: group:group@domain.com
+  hub-admin: group:org-admins@kcc.landing.systems
   #################
   # Org Policies
   #######
@@ -23,19 +39,19 @@ data:
   # org-policies/exceptions/compute-restrict-vpc-peering-except-hub-project.yaml
   # this setting MUST be changed to include the ORG ID
   project-allowed-restrict-vpc-peering: |
-    - under:organizations/ORGANIZATION_ID
+    - under:organizations/156...93
   # This list constraint defines the set of Compute Engine VM instances that are allowed to use external IP addresses, see YAML file for more info:
   # org-policies/exceptions/compute-vm-external-ip-access-except-hub-project.yaml
   # this setting MUST be changed to include the hub project ID
   project-allowed-vm-external-ip-access: |
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
   # This list constraint defines the set of VM instances that can enable IP forwarding., see YAML file for more info:
   # org-policies/exceptions/compute-vm-can-ip-forward-except-hub-project.yaml
   # this setting MUST be changed to include the hub project ID
   project-allowed-vm-can-ip-forward: |
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
-    - "projects/HUB_PROJECT_ID/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
+    - "projects/dmu-admin1-hub-kls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
   #################
   # Fortigate
   #################
@@ -46,13 +62,15 @@ data:
   # Primary
   # Having disctinct images allows one to use a Licensed Fortigate for the primary and a Pay-as-you-Go license for the secondary
   # and run the secondary just a couple of minutes each day for synching purposes thus obtaining an affordable cold standby.
-  fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  #fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  fgt-primary-image: projects/dmu-admin1-hub-kls/global/images/fortinet-fgtondemand-724-20230201-001-w-license
   # replace the word LICENSE below with the actual license value. Not required if using the Pay-as-you-Go image.
   fgt-primary-license: |
     LICENSE
   #######
   # Secondary
-  fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  #fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
+  fgt-secondary-image: projects/dmu-admin1-hub-kls/global/images/fortinet-fgtondemand-724-20230201-001-w-license
   # replace the word LICENSE below with the actual license value. Not required if using the Pay-as-you-Go image.
   fgt-secondary-license: |
     LICENSE

forgot to init - do this first

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live init hub-env --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

render

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt fn render hub-env
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.3s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    ...(102 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 2s
  Results:

apply

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live apply hub-env --reconcile-timeout=2m --output=table
installing inventory ResourceGroup CRD.
error: invalid object: "projects_dmu-admin1-hub-kls_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dmu-admin1-hub-kls -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

getting a depends error on an existing resource - the folder reference - it exists - triaging

checking it it requires the folder id

  #project-parent-folder: services-infrastructure
  project-parent-folder: "176411558066"

but fix folder.yaml manually
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure
to
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/176411558066

same thing after a render and apply

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live apply hub-env --reconcile-timeout=2m --output=table
installing inventory ResourceGroup CRD.
error: invalid object: "projects_dmu-admin1-hub-kls_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dmu-admin1-hub-kls -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/176411558066

comment out dependency - rerun

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live apply hub-env --reconcile-timeout=2m --output=table
installing inventory ResourceGroup CRD.
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-computeins  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-serviceacc  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-serviceacc  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-ext-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-int-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-transit-a  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-ext-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-int-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-transit  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-address            Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-proxy-address      Pending       Unknown                 -                                         -                                               
networking  ComputeBackendService/hub-ilb-bes         Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-fortigates-ha-  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-spokes-to-fort  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-elb-allow-health-che  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-iap-allow-rdp-to-man  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-ilb-allow-health-che  Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-managementvm-allow-s  Pending       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-fwdrule     Pending       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-proxy-fwdr  Pending       Unknown                 -                                         -                                               
networking  ComputeHTTPHealthCheck/hub-http-8008-htt  Pending       Unknown                 -                                         -                                               
networking  ComputeHealthCheck/hub-http-8008-hc       Pending       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-primary-instance  Pending       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-secondary-instan  Pending       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-management-instance   Pending       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-primary-umi  Pending       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-secondary-u  Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-external-vpc    Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-internal-vpc    Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-mgmt-vpc        Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-transit-vpc     Pending       Unknown                 -                                         -                                               
networking  ComputeRoute/hub-external-vpc-internet-e  Pending       Unknown                 -                                         -                                               
networking  ComputeRoute/hub-internal-vpc-internet-e  Pending       Unknown                 -                                         -                                               
networking  ComputeRouter/hub-nane1-external-router   Pending       Unknown                 -                                         -                                               
networking  ComputeRouterNAT/hub-nane1-external-nat   Pending       Unknown                 -                                         -                                               
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE 

nothing deployed

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live status hub-env
no resources found in the inventory

returning simpler folder

  project-parent-folder: kcc

render to adjust previous

  folderRef:
    name: 176411558066 # kpt-set: ${project-parent-folder}
    namespace: hierarchy

running

policies    ResourceManagerPolicy/compute-vm-externa  Pending       Unknown                 -                                         -                                               
projects    Project/dmu-admin1-hub-kls                Successful    InProgress              Ready                                     18s     reference Folder hierarchy/kcc is not fo
projects    Service/dmu-admin1-hub-kls-compute        Pending       Unknown                 -                                         -                                               
projects    Service/dmu-admin1-hub-kls-dns            Pending       Unknown                 -                                         -                                               

the folder is still the issue because of the missing namespace

namespace: hierarchy

but it is there

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE     READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    6h27m   True    UpToDate   6h27m
folder.resourcemanager.cnrm.cloud.google.com/clients                   6h27m   True    UpToDate   6h27m
folder.resourcemanager.cnrm.cloud.google.com/services                  6h27m   True    UpToDate   6h27m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   6h27m   True    UpToDate   6h27m

of course - kcc is not in scope of the package - returning to services-infrastructure

rerunning after render / apply

projects    Project/dmu-admin1-hub-kls                Successful    Failed                  Ready                                     5m      Update call failed: error applying desir

is IAM permissions

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live status hub-env
inventory-38012504/project.resourcemanager.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls is Failed: Update call failed: error applying desired state: summary: failed pre-requisites: missing permission on "billingAccounts/01A...99F": billing.resourceAssociations.create

switching back to local BID

015***

rerunning

projects    Project/dmu-admin1-hub-kls                Successful    InProgress              Ready                                     7m      Update in progress                      

2226: project created

dmu-admin1-hub-kls | dmu-admin1-hub-kls

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Failed                  Ready                                     26s     Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     26s     Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     26s     Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     26s     Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     25s     Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     25s     Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-int-addre  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-primary-transit-a  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-ext-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-int-add  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-fgt-secondary-transit  Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-address            Pending       Unknown                 -                                         -                                               
networking  ComputeAddress/hub-ilb-proxy-address      Pending       Unknown                 -                                         -                                               
networking  ComputeBackendService/hub-ilb-bes         Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Pending       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Pending       Unknown                 -                                         -

2233

working through failures/iam issues

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$ kpt live status hub-env
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-ext-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-ext-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-int-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-int-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-mgmt-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-mgmt-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-transit-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-transit-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-ilb-address is Current: Resource is Current
inventory-38012504/computeaddress.compute.cnrm.cloud.google.com/networking/hub-ilb-proxy-address is Current: Resource is Current
inventory-38012504/iamcustomrole.iam.cnrm.cloud.google.com/config-control/hub-fortigatesdnreader-role is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing organizations/123456789012/roles/FortigateSdnViewer: googleapi: Error 403: You don't have permission to get the role at organizations/123456789012/roles/FortigateSdnViewer.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "iam.googleapis.com",
    "metadata": {
      "permission": "iam.roles.get",
      "resource": "organizations/123456789012/roles/FortigateSdnViewer"
    },
    "reason": "IAM_PERMISSION_DENIED"
  }
]
, forbidden
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-allow-external-fwr is Current: Resource is Current
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-elb-allow-health-checks-to-fortigate-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-allow-spokes-to-fortigates-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-ilb-allow-health-checks-to-fortigate-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-allow-fortigates-ha-fwr is InProgress: reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
inventory-38012504/computeinstance.compute.cnrm.cloud.google.com/networking/hub-fgt-primary-instance is InProgress: reference ComputeDisk networking/hub-fgt-primary-log-disk is not found
inventory-38012504/computeinstance.compute.cnrm.cloud.google.com/networking/hub-fgt-secondary-instance is InProgress: reference ComputeDisk networking/hub-fgt-secondary-log-disk is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-iap-allow-rdp-to-managementvm-fwr is InProgress: reference IAMServiceAccount networking/hub-managementvm-sa is not found
inventory-38012504/computefirewall.compute.cnrm.cloud.google.com/networking/hub-managementvm-allow-ssh-https-to-fortigates-fwr is InProgress: reference IAMServiceAccount networking/hub-managementvm-sa is not found
inventory-38012504/computeinstance.compute.cnrm.cloud.google.com/networking/hub-management-instance is InProgress: reference ComputeDisk networking/hub-mgmt-data-disk is not found
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-external-logging-dnspolicy is Current: Resource is Current
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-internal-logging-dnspolicy is Current: Resource is Current
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-mgmt-logging-dnspolicy is Current: Resource is Current
inventory-38012504/dnspolicy.dns.cnrm.cloud.google.com/networking/hub-transit-logging-dnspolicy is Current: Resource is Current
inventory-38012504/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Current: Resource is Current
inventory-38012504/computerouter.compute.cnrm.cloud.google.com/networking/hub-nane1-external-router is Current: Resource is Current
inventory-38012504/computeroute.compute.cnrm.cloud.google.com/networking/hub-external-vpc-internet-egress-route is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-external-paz-snet is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-internal-paz-snet is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-mgmt-rz-snet is Current: Resource is Current
inventory-38012504/computesubnetwork.compute.cnrm.cloud.google.com/networking/hub-nane1-transit-paz-snet is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-external-vpc is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-internal-vpc is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-mgmt-vpc is Current: Resource is Current
inventory-38012504/computenetwork.compute.cnrm.cloud.google.com/networking/hub-global-transit-vpc is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-disable-serial-port-access-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-require-shielded-vm-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-load-balancer-creation-for-types-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-vpc-peering-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward-except-hub-project is Current: Resource is Current
inventory-38012504/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access-except-hub-project is Current: Resource is Current
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-serviceaccountadmin-permissions is Failed: Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/iam.serviceAccountAdmin serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountAdmin serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-serviceaccountuser-permissions is Failed: Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/iam.serviceAccountUser serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/iam.serviceAccountUser serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/networking-sa-computeinstanceadmin-permissions is Failed: Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/hub-admin-iaptunnelresourceaccessor-permissions is Current: Resource is Current
inventory-38012504/iampolicymember.iam.cnrm.cloud.google.com/config-control/hub-admin-computeinstanceadmin-permissions is Current: Resource is Current
inventory-38012504/project.resourcemanager.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls is Current: Resource is Current
inventory-38012504/service.serviceusage.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls-compute is Current: Resource is Current
inventory-38012504/service.serviceusage.cnrm.cloud.google.com/projects/dmu-admin1-hub-kls-dns is Current: Resource is Current
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls)$

working out iam permissions issues

fmichaelobrien commented 10 months ago

working additions main not 0.2.0 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/releases/tag/solutions%2Fproject%2Fhub-env%2F0.2.0

custom FortigateSdnViewer role is in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigate/custom-role.yaml#L27

add
organizations/123456789012/roles/FortigateSdnViewer

and IAMServiceAccount networking/hub-managementvm-sa

todo:

add client-landing-zone on top of existing core-landing-zone before hub-env

Package Inventory

obriensystems commented 10 months ago

add packages

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/client-landing-zone@main
Package "client-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 + ea2e57f...10ca23d main       -> origin/main  (forced update)
Adding package "solutions/client-landing-zone".

Fetched 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/client-setup@main
Package "client-setup":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
Adding package "solutions/client-setup".

Fetched 1 package(s).

obriensystems commented 9 months ago

Current status (deploying hub) - move from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/445 kcc.landing.zone root@cloudshell:~/kcc-kls/lz-20230803$ ls client-landing-zone client-setup core-landing-zone hub-env setters.yaml root@cloudshell:~/kcc-kls/lz-20230803$

obriensystems commented 9 months ago

Known Issues Workarounds

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigates.md#known-issues
turn down security on the management VPC - turn back on public access for googleapis.com public access without putting in a private service connect endpoint like
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/468

obriensystems commented 9 months ago

restarting hub-env adjustment

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render hub-env
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.1s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    ...(102 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 1.7s
  Results:
    [info]: no matches

Successfully executed 2 function(s) in 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$  kpt live apply hub-env --reconcile-timeout=2m --output=table

config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Failed                  Ready                                     1167h   Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Skipped       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-int-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-transit-a  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-ext-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-int-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-transit  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-address            Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-proxy-address      Successful    Current                 Ready                                     1167h   Resource is Current                     
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol  Successful    Failed                  Ready                                     1167h   Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Skipped       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     1167h   Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1167h   Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-int-addre  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-transit-a  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-ext-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-int-add  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-transit  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-address            Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeAddress/hub-ilb-proxy-address      Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeBackendService/hub-ilb-bes         Pending       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Skipped       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeFirewall/hub-allow-fortigates-ha-  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-allow-spokes-to-fort  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-elb-allow-health-che  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-iap-allow-rdp-to-man  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-ilb-allow-health-che  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-managementvm-allow-s  Successful    InProgress              Ready                                     1167h   reference IAMServiceAccount networking/h
networking  ComputeForwardingRule/hub-ilb-fwdrule     Pending       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-proxy-fwdr  Pending       Unknown                 -                                         -                                               
networking  ComputeHTTPHealthCheck/hub-http-8008-htt  Skipped       Unknown                 -                                         -                                               
networking  ComputeHealthCheck/hub-http-8008-hc       Skipped       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-primary-instance  Successful    InProgress              Ready                                     1167h   reference ComputeDisk networking/hub-fgt
networking  ComputeInstance/hub-fgt-secondary-instan  Successful    InProgress              Ready                                     1167h   reference ComputeDisk networking/hub-fgt
networking  ComputeInstance/hub-management-instance   Successful    InProgress              Ready                                     1167h   reference ComputeDisk networking/hub-mgm
networking  ComputeInstanceGroup/hub-fgt-primary-umi  Pending       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-secondary-u  Pending       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-external-vpc    Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeNetwork/hub-global-internal-vpc    Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeNetwork/hub-global-mgmt-vpc        Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeNetwork/hub-global-transit-vpc     Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeRoute/hub-external-vpc-internet-e  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeRoute/hub-internal-vpc-internet-e  Pending       Unknown                 -                                         -                                               
networking  ComputeRouter/hub-nane1-external-router   Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeRouterNAT/hub-nane1-external-nat   Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-external-paz  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-internal-paz  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-mgmt-rz-snet  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-transit-paz-  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  ComputeTargetPool/hub-elb-pool            Pending       Unknown                 -                                         -                                               
networking  DNSPolicy/hub-external-logging-dnspolicy  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  DNSPolicy/hub-internal-logging-dnspolicy  Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  DNSPolicy/hub-mgmt-logging-dnspolicy      Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  DNSPolicy/hub-transit-logging-dnspolicy   Successful    Current                 Ready                                     1167h   Resource is Current                     
networking  IAMPolicyMember/hub-admin-serviceaccount  Skipped       Unknown                 -                                         -                                               
networking  IAMServiceAccount/hub-fortigatesdn-sa     Skipped       Unknown                 -                                         -                                               
networking  IAMServiceAccount/hub-managementvm-sa     Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     1167h   Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     1167h   Resource is Current                     
projects    Project/dmu-admin1-hub-kls                Successful    Current                 Ready                                     1167h   Resource is Current                     
projects    Service/dmu-admin1-hub-kls-compute        Successful    Current                 Ready                                     1167h   Resource is Current                     
projects    Service/dmu-admin1-hub-kls-dns            Successful    Current                 Ready                                     1167h   Resource is Current

obriensystems commented 8 months ago

Restarting clean org fortigate install for monday obrien.industries

Deployment change - we will switch to an in-place kpt render (right in the github repo) - so we can track changes

see fine tuning of the wiki documentation in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#quickstart

Install the KCC cluster and minimal set of Landing Zone packages

0 overview https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/README.md#quickstart
1 see kcc cluster install script in https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh
2 see landing zone install docs in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/README.md#2-create-your-landing-zone

Scenarios

workflow 1 - shut down cluster only (create kcc cluster - create lz (render kpt lz packages) - delete kcc cluster - recreate cluster (automatically acquire gcp resources)
workflow 2 - shut down lz and cluster (create kcc cluster - create lz (render kpt lz packages) - delete lz packages - delete kcc cluster

prereq = billing quota above 5, liens commented in the code, org polices (gatekeeper) omitted

create kcc cluster = https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L107

gcloud anthos config controller get-credentials $CLUSTER  --location $REGION

but use https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh

gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET" --master-ipv4-cidr-block="172.16.0.128/28" --full-management "${args[@]}"
else

create lz = https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L165

# packages core-landing-zone, client-landing-zone, client-setup, project/hub-env
kpt live init core-landing-zone --namespace config-control --force
kpt fn render core landing-zone
kpt live apply core-landing-zone --reconcile-timeout=2m --output=table

delete lz = (including liens) https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L198C8-L198C8

gcloud alpha resource-manager liens delete $NONPROD_LIEN # all 3
kpt live destroy core-landing-zone

delete kcc cluster = https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L206

gcloud anthos config controller delete --location $REGION $CLUSTER --quiet

Plan

clean org
install kcc cluster - via script
install 4 lz packages via script (to be re-written from kcc lz 1 version at https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L164 under https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/192

    1  export EMAIL=michael@obrien.industries
    2  gcloud organizations get-iam-policy roles/resourcemanager.organizationAdmin --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
    3  gcloud organizations list
    4  gcloud organizations list | grep ID
    5  gcloud organizations list --format="get(name)"
    6  export DOMAIN=obrien.industries
    7  ORG_ID=$(gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN)
    8  echo $ORG_ID
    9  gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN
   10  export ORG_ID=459065442144
   11  gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
   12  gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$EMAIL --role=iam.serviceAccountTokenCreator
   13  gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$EMAIL --role=roles/iam.serviceAccountTokenCreator
   14  gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/iam.serviceAccountTokenCreator
   15  gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/billing.projectManager
   16  mkdir kcc-oi
   17  cd kcc-oi
   18  mkdir github
   19  mkdir kpt
   20  cd github
   21  git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git
   22  git clone https://github.com/ssc-spc-ccoe-cei/gcp-tools.git
   23  cd gcp-tools/scripts/bootstrap/
   24  cp .env.sample kcc.env
   25  export PROJECT_ID=kcc-oi
   26  gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
   27  gcloud config set project "${PROJECT_ID}"
   28  echo $ORG_ID
   29  export ROOT_FOLDER=kcc
   30  gcloud resource-manager folders create --display-name=$ROOT_FOLDER --organization=$ORG_ID
   31  export BILLING_ID=$(gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' | sed 's/.*\///')
   32  echo $BILLING_ID
   33  gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' 
   34  echo $PROJECT_ID
   35  export BILLING_ID=014479-806359-2F5F85
   36  gcloud beta billing projects link "$PROJECT_ID" --billing-account "$BILLING_ID"
   37  ls
   38  chmod 777 setup-kcc.sh 
   39  ./setup-kcc.sh -af kcc.env
   40  history
   41  gcloud config set project kcc-oi
   42  cd kcc-oi/
   43  ls
   44  cd kpt/
   45  PACKAGE="solutions/gatekeeper-policies"
   46  VERSION=$(curl -s $URL | jq -r ".\"$PACKAGE\"")
   47  URL=https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   48  VERSION=$(curl -s $URL | jq -r ".\"$PACKAGE\"")
   49  curl -s $URL
   50  https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   51  curl -s $URL | jq -r ".\"$PACKAGE\""
   52  VERSION=main
   53  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
   54  PACKAGE="solutions/core-landing-zone"
   55  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
   56  gcloud organizations list
   57  kpt live init core-landing-zone --namespace config-control
   58  kpt fn render core-landing-zone
   59  gcloud config set project kcc-oi
   60  cd kcc-oi/
   61  cd kpt/
   62  ls
   63  kpt fn render core-landing-zone
   64  gcloud config set project kcc-oi
   65  gcloud config set project kcc-oi-cluster
   66  kubectl edit validatingwebhookconfiguration/gatekeeper-validating-webhook-configuration
   67  kubectl get nodes
   68  gcloud config set project kcc-oi
   69  kubectl get nodes
   70  kubectl get pods --all-namespaces
   71  gcloud anthos config controller get-credentials krmapihost-kcc-oi  --location northamerica-northeast1
   72  gcloud config set project kcc-oi-cluster
   73  gcloud anthos config controller get-credentials krmapihost-kcc-oi  --location northamerica-northeast1
   74  gcloud anthos config controller get-credentials kcc-oi  --location northamerica-northeast1
   75  kubens config-control
   76  kubectl get pods --all-namespaces
   77  cd kcc-oi/kpt/
   78  ls
   79  kpt live apply core-landing-zone --reconcile-timeout=2m --output=table
   80  gcloud config set project kcc-oi-cluster
   81  kubectl get pods
   82  ls
   83  ls -la
   84  ls -la .kube/
   85  cat .kube/config
   86  kubectl get nodes
   87  history
   88  gcloud config set project kcc-oi
   89  cd kcc-oi/github/
   90  cd gcp-tools/scripts/bootstrap/
   91  ./setup-kcc.sh -afp kcc.env
   92  kubectl get nodes
   93  history

gcp-tools

michael@cloudshell:~/kcc-oi/github/gcp-tools (kcc-oi)$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   scripts/bootstrap/setup-kcc.sh

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        scripts/bootstrap/kcc.env

kcc.env
export CLUSTER=kcc-oi2
export REGION=northamerica-northeast1
export PROJECT_ID=kcc-oi2-cluster
export LZ_FOLDER_NAME=kcc-lz-20230928b
export NETWORK=kcc-oi2-vpc
export SUBNET=kcc-oi2-sn
export ORG_ID=459065442144
export ROOT_FOLDER_ID=96269513997
export BILLING_ID=014479-806359-2F5F85
#export GIT_USERNAME=obriensystems
#export CONFIG_SYNC_REPO=<Repo for Config Sync> # tierX repo URL
#export CONFIG_SYNC_VERSION='HEAD'
#export CONFIG_SYNC_DIR=<Directory for config sync repo which syncs> # Should default to csync/deploy/<env>

refresh repo with main 20231019

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git pull
remote: Enumerating objects: 202, done.
remote: Counting objects: 100% (202/202), done.
remote: Compressing objects: 100% (99/99), done.
remote: Total 202 (delta 126), reused 163 (delta 102), pack-reused 0
Receiving objects: 100% (202/202), 80.52 KiB | 8.95 MiB/s, done.
Resolving deltas: 100% (126/126), completed with 38 local objects.
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   8370e06..6260c22  main                                              -> origin/main
 * [new branch]      dependabot/go_modules/cli/golang.org/x/net-0.17.0 -> origin/dependabot/go_modules/cli/golang.org/x/net-0.17.0
 * [new branch]      gh540-fmichaelobrien-temp-fix-kpt-readme          -> origin/gh540-fmichaelobrien-temp-fix-kpt-readme
 * [new branch]      gh563-fix-cleanup-tier1                           -> origin/gh563-fix-cleanup-tier1
 * [new branch]      https-elb-example                                 -> origin/https-elb-example
 * [new tag]         solutions/client-landing-zone/0.4.6               -> solutions/client-landing-zone/0.4.6
 * [new tag]         solutions/client-landing-zone/0.4.7               -> solutions/client-landing-zone/0.4.7
 * [new tag]         solutions/client-setup/0.6.1                      -> solutions/client-setup/0.6.1
Updating 8370e06..6260c22
Fast-forward
 .release-please-manifest.json                                                                        |   4 +-
 docs/landing-zone-v2/README.md                                                                       |   7 +-
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/README.md                |  27 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/address.yaml             |  27 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/backend-service.yaml     |  60 ++++++++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/elb.yaml                 |  27 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/firewall.yaml            |  38 ++++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/forwarding-rule.yaml     |  34 ++++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/health-check.yaml        |  32 +++++
 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/target-https-proxy.yaml  |  33 +++++
 examples/landing-zone-v2/setters.yaml                                                                |   2 +
 solutions/client-landing-zone/CHANGELOG.md                                                           |  14 +++
 solutions/client-landing-zone/README.md                                                              |  10 +-
 solutions/client-landing-zone/client-folder/firewall-policy/policy.yaml                              |   9 +-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/defaults.yaml                      |  17 ++-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/iap.yaml                           |   6 +-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/lb-health-checks.yaml              |   6 +-
 solutions/client-landing-zone/client-folder/firewall-policy/rules/os-updates.yaml                    |  10 +-
 solutions/client-landing-zone/client-folder/folder-iam.yaml                                          |   3 +-
 solutions/client-landing-zone/client-folder/folder-sink.yaml                                         |   8 +-
 .../client-folder/standard/applications-infrastructure/firewall-policy/policy.yaml                   |   7 +-
 .../client-folder/standard/applications-infrastructure/firewall-policy/rules/defaults.yaml           |  17 ++-
 .../client-folder/standard/applications-infrastructure/firewall-policy/rules/iap.yaml                |   6 +-
 .../client-folder/standard/applications-infrastructure/firewall-policy/rules/lb-health-checks.yaml   |   6 +-
 .../client-folder/standard/applications-infrastructure/host-project/network/firewall.yaml            |  46 +++++--
 .../standard/applications-infrastructure/host-project/network/psc/google-apis/firewall.yaml          |   3 +-
 .../client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml              |   5 +-
 .../client-folder/standard/applications-infrastructure/host-project/network/vpc.yaml                 |   2 +-
 solutions/client-landing-zone/client-folder/standard/firewall-policy/policy.yaml                     |   5 +-
 solutions/client-landing-zone/client-folder/standard/firewall-policy/rules/network-isolation.yaml    |  11 +-
 solutions/client-landing-zone/logging-project/cloud-logging-bucket.yaml                              |   8 +-
 solutions/client-landing-zone/logging-project/project-iam.yaml                                       |   3 +-
 solutions/client-landing-zone/securitycontrols.md                                                    | 326 +++++++++++++++++++++++++++++++++++++++++++++++---
 solutions/client-landing-zone/setters.yaml                                                           |  24 ++--
 solutions/client-setup/CHANGELOG.md                                                                  |   7 ++
 35 files changed, 764 insertions(+), 86 deletions(-)
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/README.md
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/address.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/backend-service.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/elb.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/firewall.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/forwarding-rule.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/health-check.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/https-external-load-balancer/target-https-proxy.yaml

create dev branch

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git pull
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * [new branch]      gh446-hub  -> origin/gh446-hub
Already up to date.

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git checkout gh446-hub
Branch 'gh446-hub' set up to track remote branch 'gh446-hub' from 'origin'.
Switched to a new branch 'gh446-hub'

add setters.yaml changes

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git status
On branch gh446-hub
Your branch is up to date with 'origin/gh446-hub'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   solutions/core-landing-zone/setters.yaml

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git add solutions/core-landing-zone/setters.yaml 
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit (kcc-oi)$ git commit -m "#446 - add clz setters.yaml"

undeploy any lz packages up

deploy changes - clz package of 4

push to remote repo

write part 2 lz sh script

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -c false -l false -r false -d false -p kcc-oi2-cluster

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/567

Delete cluster only

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -c false -l false -r false -d true -p kcc-oi2-cluster
Date: Fri 20 Oct 2023 01:37:16 AM UTC
Timestamp: 1697765836
running with: -b kcc-oi -u oi -c false -l false -r false -d true -p kcc-oi2-cluster
Deleting cluster kcc-oi2 in region northamerica-northeast1
Delete Cluster kcc-oi2 in region northamerica-northeast1
Delete request issued for: [kcc-oi2]
Waiting for operation [projects/kcc-oi2-cluster/locations/northamerica-northeast1/operations/operation-1697765858524-6081beae8be0d-93150922-7d4fd45a] to com
plete...done.                                                                                                                                               
Deleted instance [kcc-oi2].
Cluster delete time: 405 sec
Total Duration: 425 sec
Date: Fri 20 Oct 2023 01:44:22 AM UTC
Timestamp: 1697766262
Updated property [core/project].
Switched back to boot project kcc-oi
**** Done ****

Recreate KCC cluster

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ ./setup.sh -b kcc-oi -u oi -c true -l false -r false -d false -p kcc-oi2-cluster
Date: Fri 20 Oct 2023 01:55:26 AM UTC
Timestamp: 1697766926
running with: -b kcc-oi -u oi -c true -l false -r false -d false -p kcc-oi2-cluster

Reusing project: kcc-oi2-cluster
Updated property [core/project].
Creating Anthos KCC autopilot cluster kcc-oi2 in region northamerica-northeast1 in subnet kcc-oi2-sn off VPC kcc-oi2-vpc
Create request issued for: [kcc-oi2]
Waiting for operation [projects/kcc-oi2-cluster/locations/northamerica-northeast1/operations/operation-1697766931672-6081c2adface2-48f8ae7f-81f8305c] to com
plete...working..

plete...done.                                                                                                                                               
Created instance [kcc-oi2].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi2.
Cluster create time: 1105 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi2.
Context "gke_kcc-oi2-cluster_northamerica-northeast1_krmapihost-kcc-oi2" modified.
Active namespace is "config-control".
List Clusters:
NAME: kcc-oi2
LOCATION: northamerica-northeast1
STATE: RUNNING
Total Duration: 1111 sec
Date: Fri 20 Oct 2023 02:13:58 AM UTC
Timestamp: 1697768038
Updated property [core/project].
Switched back to boot project kcc-oi

deploy changes - clz package of 4

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ ./setup.sh -b kcc-oi -u oi -c false -l true -r false -d false -p kcc-oi2-cluster

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/logging                         Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/networking                      Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/policies                        Successful    Current                 <None>                                    4m      Resource is current                     
            Namespace/projects                        Successful    Current                 <None>                                    4m      Resource is current                     
config-con  IAMCustomRole/gke-firewall-admin          Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier2-dnsrecord-admin       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier2-vpcpeering-admin      Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-dnsrecord-admin       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-firewallrule-admin    Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-subnetwork-admin      Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier3-vpcsc-admin           Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMCustomRole/tier4-secretmanager-admin   Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/logging-sa-workload-ide  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/networking-sa-workload-  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/policies-sa-workload-id  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPartialPolicy/projects-sa-workload-id  Successful    InProgress              Ready                                     4m      reference IAMServiceAccount config-contr
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/config-control-sa-manage  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/config-control-sa-orgrol  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-dns-permis  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-networkadm  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-security-p  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-service-co  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-servicedir  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-billinguser-  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectcreat  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectdelet  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectiamad  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-projectmover  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMPolicyMember/projects-sa-serviceusage  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  IAMServiceAccount/gatekeeper-admin-sa     Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/hierarchy-sa            Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/logging-sa              Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/networking-sa           Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/policies-sa             Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  IAMServiceAccount/projects-sa             Successful    Failed                  Ready                                     4m      Update call failed: error applying desir
config-con  Service/kcc-oi-cluster-accesscontextmana  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  Service/kcc-oi-cluster-cloudbilling       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  Service/kcc-oi-cluster-cloudresourcemana  Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
config-con  Service/kcc-oi-cluster-serviceusage       Successful    Failed                  Ready                                     4m      Update call failed: error fetching live 
gatekeeper  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    4m      status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
hierarchy   RoleBinding/allow-folders-resource-refer  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Successful    Current                 <None>                                    2m      Resource is current                     
hierarchy   Folder/audits                             Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
hierarchy   Folder/clients                            Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
hierarchy   Folder/services                           Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
hierarchy   Folder/services-infrastructure            Successful    Failed                  Ready                                     2m      Update call failed: error applying desir
logging     ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
logging     LoggingLogBucket/platform-and-component-  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogBucket/security-log-bucket-oi   Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/logging-project-oi-securi  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-disa  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-plat  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     RoleBinding/allow-logging-resource-refer  Successful    Current                 <None>                                    2m      Resource is current                     
networking  ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
networking  DNSManagedZone/dns-project-oi-standard-c  Skipped       Unknown                 -                                         -                                               
policies    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
policies    ResourceManagerPolicy/compute-disable-gu  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-disable-ne  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-disable-se  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-disable-vp  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-require-os  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-require-sh  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-require-sh  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-restrict-s  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-skip-defau  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/essentialcontacts-  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/gcp-restrict-resou  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/iam-allowed-policy  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/iam-disable-servic  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/sql-restrict-publi  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/storage-public-acc  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
policies    ResourceManagerPolicy/storage-uniform-bu  Successful    Failed                  Ready                                     2m      Update call failed: error fetching live 
projects    ConfigConnectorContext/configconnectorco  Successful    Current                 <None>                                    2m      status.healthy is true                  
projects    IAMAuditConfig/logging-project-data-acce  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/mgmt-project-cluster-pl  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/platform-and-component-  Skipped       Unknown                 -                                         -                                               
projects    IAMPartialPolicy/security-log-bucket-wri  Skipped       Unknown                 -                                         -                                               
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    2m      Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    2m      Resource is current                     
projects    RoleBinding/allow-projects-resource-refe  Successful    Current                 <None>                                    2m      Resource is current                     
projects    Project/dns-project-oi                    Skipped       Unknown                 -                                         -                                               
projects    Project/logging-project-oi                Successful    InProgress              Ready                                     2m      reference Folder hierarchy/audits is not
projects    Service/dns-project-oi-dns                Skipped       Unknown                 -                                         -

wait for cnrm workloads to come up - 5 min first

triage - folder errors

ierarchy   Folder/audits                             Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir
hierarchy   Folder/clients                            Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir
hierarchy   Folder/services                           Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir
hierarchy   Folder/services-infrastructure            Skipped       Failed                  Ready                                     13m     Update call failed: error applying desir

switched setters.yaml
management-project-id: kcc-oi-cluster
management-project-id: kcc-oi2-cluster

delete partial lz

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ ./setup.sh -b kcc-oi -u oi -c false -l false -r true -d false -p kcc-oi2-cluster

resource-group-system             resource-group-controller-manager-7dbf5b5766-s9sr7         2/2     Running   0             32m
deleting lz on kcc-oi2 in region northamerica-northeast1
delete phase started
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage delete successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager delete successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling delete successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager delete successful
delete phase finished
reconcile phase started
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 4 attempted, 4 successful, 0 skipped, 0 failed
reconcile result: 4 attempted, 4 successful, 0 skipped, 0 failed, 0 timed out
Total Duration: 21 sec
Date: Fri 20 Oct 2023 02:40:22 AM UTC
Timestamp: 1697769622
Updated property [core/project].
Switched back to boot project kcc-oi

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl delete gcp --all
iamcustomrole.iam.cnrm.cloud.google.com "gke-firewall-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier2-dnsrecord-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier2-vpcpeering-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-dnsrecord-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-firewallrule-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-subnetwork-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-vpcsc-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier4-secretmanager-admin" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "gatekeeper-admin-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "hierarchy-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "logging-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "networking-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "policies-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "projects-sa-workload-identity-binding" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-management-project-editor-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-management-project-serviceaccountadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-orgroleadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "gatekeeper-admin-sa-metric-writer-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "hierarchy-sa-folderadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "logging-sa-bigqueryadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "logging-sa-logadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-dns-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-networkadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-security-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-service-control-org-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-servicedirectoryeditor-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-xpnadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "policies-sa-orgpolicyadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-billinguser-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectcreator-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectdeleter-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectiamadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectmover-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-serviceusageadmin-permissions" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "gatekeeper-admin-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "hierarchy-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "logging-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "networking-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "policies-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "projects-sa" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-accesscontextmanager" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-cloudbilling" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-cloudresourcemanager" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-cluster-serviceusage" deleted

adjust setters.yaml

delete/recreate cluster

 dns-name: "obrien.industries."
  management-project-number: "180205379034"

Triage pod failures

kubectl get gcp
kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa

  Warning  UpdateFailed  36s (x12 over 12m)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Caller does not have required permission to use project kcc-oi2-cluster. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=kcc-oi2-cluster and then retry. Propagation of the new permission may take a few minutes.

Adding security Admin to super admin (is in kcc.landing.systems) and adding service usage consumer role

2400: got it - should not have commented out the gKE service account - the yakima one

 # Assign Permissions to the KCC Service Account - will need a currently running kcc cluster
#  export SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"

#  echo "SA_EMAIL: ${SA_EMAIL}"
#  ROLES=("roles/bigquery.dataEditor" "roles/serviceusage.serviceUsageAdmin" "roles/logging.configWriter" "roles/resourcemanager.projectIamAdmin" "roles/resourcemanager.organizationAdmin" "roles/iam.organizationRoleAdmin" "roles/compute.networkAdmin" "roles/resourcemanager.folderAdmin" "roles/resourcemanager.projectCreator" "roles/resourcemanager.projectDeleter" "roles/resourcemanager.projectMover" "roles/iam.securityAdmin" "roles/orgpolicy.policyAdmin" "roles/serviceusage.serviceUsageConsumer" "roles/billing.user" "roles/accesscontextmanager.policyAdmin" "roles/compute.xpnAdmin" "roles/iam.serviceAccountAdmin" "roles/serviceusage.serviceUsageConsumer" "roles/logging.admin") 
#  for i in "${ROLES[@]}" ; do
    # requires iam.securityAdmin
    #ROLE=`gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$SA_EMAIL" --flatten="bindings[].members" --format="table(bindings.role)" | grep $i`
    #echo $ROLE
    #if [ -z "$ROLE" ]; then
#        echo "Applying role $i to $SA_EMAIL"
#        gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$SA_EMAIL --role=$i --quiet > /dev/null 1>&1
    #else
    #    echo "Role $i already set on $USER"
    #fi
#  done

running
kube-system                       netd-w5m97                                                 1/1     Running   0               69m
kube-system                       node-local-dns-5gfds                                       1/1     Running   0               73m
kube-system                       node-local-dns-flq8w                                       1/1     Running   0               69m
kube-system                       node-local-dns-krw4v                                       1/1     Running   0               2m19s
kube-system                       node-local-dns-mm2k8                                       1/1     Running   0               69m
kube-system                       node-local-dns-pqrqb                                       1/1     Running   0               6m47s
kube-system                       pdcsi-node-cjjs9                                           2/2     Running   0               69m
kube-system                       pdcsi-node-hdz4x                                           2/2     Running   0               73m
kube-system                       pdcsi-node-ntc5r                                           2/2     Running   0               2m23s
kube-system                       pdcsi-node-svd24                                           2/2     Running   0               6m51s
kube-system                       pdcsi-node-tpmmv                                           2/2     Running   0               69m
resource-group-system             resource-group-controller-manager-7dbf5b5766-z9ncd         2/2     Running   0               5m19s
SA_EMAIL: service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Applying role roles/bigquery.dataEditor to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/serviceusage.serviceUsageAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/logging.configWriter to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectIamAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.organizationAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/iam.organizationRoleAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/compute.networkAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.folderAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectCreator to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectDeleter to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/resourcemanager.projectMover to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/iam.securityAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/orgpolicy.policyAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/serviceusage.serviceUsageConsumer to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/billing.user to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/accesscontextmanager.policyAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/compute.xpnAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/iam.serviceAccountAdmin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/serviceusage.serviceUsageConsumer to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Applying role roles/logging.admin to service-180205379034@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
kpt live init

raised https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/568

services coming up now

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp
NAME                                                                AGE     READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          7m27s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       7m27s   True    UpToDate       77s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      7m26s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       7m26s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    7m26s   False   UpdateFailed   7m26s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      7m26s   True    UpToDate       77s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           7m26s   False   UpdateFailed   7m25s
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   7m25s   False   UpdateFailed   7m25s

NAME                                                                                       AGE     READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   7m26s   False   DependencyNotReady   7m26s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          7m26s   False   DependencyNotFound   7m26s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            7m26s   False   DependencyNotFound   7m26s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         7m26s   False   DependencyNotReady   7m25s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           7m25s   False   DependencyNotReady   7m25s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           7m25s   False   DependencyNotFound   7m25s

NAME                                                                                                             AGE     READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                7m25s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   7m25s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             7m24s   True    UpToDate       76s
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          7m24s   True    UpToDate       63s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   7m24s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   7m24s   False   UpdateFailed   7m24s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        7m24s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       7m23s   False   UpdateFailed   7m23s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     7m23s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 7m22s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    7m22s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 7m22s   False   UpdateFailed   7m22s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 7m22s   False   UpdateFailed   7m21s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                7m21s   False   UpdateFailed   7m21s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   7m21s   False   UpdateFailed   7m20s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              7m20s   False   UpdateFailed   7m20s

NAME                                                              AGE     READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   7m20s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          7m20s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            7m19s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         7m19s   False   UpdateFailed   7m19s
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           7m19s   False   UpdateFailed   7m17s
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           7m18s   False   UpdateFailed   7m17s

NAME                                                                              AGE     READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager   7m19s   True    UpToDate   30s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling           7m19s   True    UpToDate   2m44s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager   7m18s   True    UpToDate   41s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage           7m18s   True    UpToDate   2m44s

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp | grep UpToDate | wc -l
40
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp | grep UpToDate | wc -l
44

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi2-cluster)$ kubectl get gcp
NAME                                                                AGE     READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          4m25s   True    UpToDate   4m25s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       4m25s   True    UpToDate   4m25s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      4m25s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       4m25s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    4m24s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      4m24s   True    UpToDate   4m24s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           4m24s   True    UpToDate   4m23s
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   4m24s   True    UpToDate   4m23s

NAME                                                                                       AGE     READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   4m23s   True    UpToDate   4m16s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          4m23s   True    UpToDate   4m15s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            4m23s   True    UpToDate   4m16s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         4m23s   True    UpToDate   4m14s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           4m23s   True    UpToDate   4m14s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           4m22s   True    UpToDate   3m31s

NAME                                                                                                             AGE     READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                4m22s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   4m22s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             4m22s   True    UpToDate   4m16s
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          4m22s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   4m21s   True    UpToDate   3m52s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   4m21s   True    UpToDate   3m43s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        4m21s   True    UpToDate   4m9s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          4m20s   True    UpToDate   3m42s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 4m20s   True    UpToDate   3m42s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     4m20s   True    UpToDate   4m5s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          4m20s   True    UpToDate   3m56s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       4m20s   True    UpToDate   4m4s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     4m19s   True    UpToDate   3m56s
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 4m19s   True    UpToDate   3m55s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    4m19s   True    UpToDate   3m19s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 4m19s   True    UpToDate   3m23s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 4m19s   True    UpToDate   3m5s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                4m19s   True    UpToDate   3m4s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   4m18s   True    UpToDate   3m4s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              4m18s   True    UpToDate   3m4s

NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   4m18s   True    UpToDate   4m17s
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          4m18s   True    UpToDate   4m16s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            4m17s   True    UpToDate   4m16s
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         4m17s   True    UpToDate   4m16s
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           4m17s   True    UpToDate   4m15s
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           4m17s   True    UpToDate   3m32s

NAME                                                                              AGE     READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-accesscontextmanager   4m20s   True    UpToDate   4m15s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudbilling           4m19s   True    UpToDate   4m16s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-cloudresourcemanager   4m19s   True    UpToDate   4m16s
service.serviceusage.cnrm.cloud.google.com/kcc-oi2-cluster-serviceusage           4m19s   True    UpToDate   4m16s

obriensystems commented 8 months ago

https://github.com/ssc-spc-ccoe-cei/gcp-tools/pull/53 and https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/568

editupdate: found them in the new 2nd script

https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh#L35

Issue is that the access script assumes rootsync usage - it leaves out the kpt optionI recommend we put the yakima service account role additions back to the generic setup script.

obriensystems commented 8 months ago

automation test target env root at landing.systems dev target obrien.enginnering partial fortigate kcc.landing.systems pull/run https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh446-hub

skip https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/296#issuecomment-1450681459 move to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/445#issuecomment-1669512029

need to be done manually

verify org level sa roles in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/solutions/core-landing-zone/0.3.0/docs/landing-zone-v2/README.md#1-complete-the-bootstrap-procedure create landing-zone folder create kcc-boot-ls project in the folder

switch to automation

repo already cloned
mkdir kpt folder at the root
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit$ cd solutions/
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions$ gcloud config set project kcc-boot-ls
Updated property [core/project].
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ 
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ mkdir ../../kpt

push super admin changes in #570

obriensystems commented 8 months ago

add fix for hub-env setters.yaml missing org-id in #573 should unblock anything under the SDN custom role

NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
config-con  IAMCustomRole/hub-fortigatesdnreader-rol                Failed                  Ready                                     1602h   Update call failed: error fetching live 
config-con  IAMPolicyMember/fortigatesdn-sa-fortigat  Skipped       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hub-admin-computeinstanc  Successful    Current                 Ready                                     1602h   Resource is Current                     
config-con  IAMPolicyMember/hub-admin-iaptunnelresou  Successful    Current                 Ready                                     1602h   Resource is Current                     
config-con  IAMPolicyMember/networking-sa-computeins  Successful    Failed                  Ready                                     1602h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1602h   Update call failed: error setting policy
config-con  IAMPolicyMember/networking-sa-serviceacc  Successful    Failed                  Ready                                     1602h   Update call failed: error setting policy
networking  ComputeAddress/hub-fgt-primary-ext-addre  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-int-addre  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-mgmt-addr  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-primary-transit-a  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-ext-add  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-int-add  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-mgmt-ad  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-fgt-secondary-transit  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-ilb-address            Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeAddress/hub-ilb-proxy-address      Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeBackendService/hub-ilb-bes         Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-primary-log-disk      Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-fgt-secondary-log-disk    Skipped       Unknown                 -                                         -                                               
networking  ComputeDisk/hub-mgmt-data-disk            Skipped       Unknown                 -                                         -                                               
networking  ComputeFirewall/hub-allow-external-fwr    Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeFirewall/hub-allow-fortigates-ha-  Successful    InProgress              Ready                                     1602h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-allow-spokes-to-fort  Successful    InProgress              Ready                                     1602h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-elb-allow-health-che  Successful    InProgress              Ready                                     1602h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-iap-allow-rdp-to-man  Successful    InProgress              Ready                                     1602h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-ilb-allow-health-che  Successful    InProgress              Ready                                     1602h   reference IAMServiceAccount networking/h
networking  ComputeFirewall/hub-managementvm-allow-s  Successful    InProgress              Ready                                     1602h   reference IAMServiceAccount networking/h
networking  ComputeForwardingRule/hub-ilb-fwdrule     Skipped       Unknown                 -                                         -                                               
networking  ComputeForwardingRule/hub-ilb-proxy-fwdr  Skipped       Unknown                 -                                         -                                               
networking  ComputeHTTPHealthCheck/hub-http-8008-htt  Skipped       Unknown                 -                                         -                                               
networking  ComputeHealthCheck/hub-http-8008-hc       Skipped       Unknown                 -                                         -                                               
networking  ComputeInstance/hub-fgt-primary-instance  Successful    InProgress              Ready                                     1602h   reference ComputeDisk networking/hub-fgt
networking  ComputeInstance/hub-fgt-secondary-instan  Successful    InProgress              Ready                                     1602h   reference ComputeDisk networking/hub-fgt
networking  ComputeInstance/hub-management-instance   Successful    InProgress              Ready                                     1602h   reference ComputeDisk networking/hub-mgm
networking  ComputeInstanceGroup/hub-fgt-primary-umi  Skipped       Unknown                 -                                         -                                               
networking  ComputeInstanceGroup/hub-fgt-secondary-u  Skipped       Unknown                 -                                         -                                               
networking  ComputeNetwork/hub-global-external-vpc    Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeNetwork/hub-global-internal-vpc    Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeNetwork/hub-global-mgmt-vpc        Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeNetwork/hub-global-transit-vpc     Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeRoute/hub-external-vpc-internet-e  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeRoute/hub-internal-vpc-internet-e  Skipped       Unknown                 -                                         -                                               
networking  ComputeRouter/hub-nane1-external-router   Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeRouterNAT/hub-nane1-external-nat   Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-external-paz  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-internal-paz  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-mgmt-rz-snet  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeSubnetwork/hub-nane1-transit-paz-  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  ComputeTargetPool/hub-elb-pool            Skipped       Unknown                 -                                         -                                               
networking  DNSPolicy/hub-external-logging-dnspolicy  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  DNSPolicy/hub-internal-logging-dnspolicy  Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  DNSPolicy/hub-mgmt-logging-dnspolicy      Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  DNSPolicy/hub-transit-logging-dnspolicy   Successful    Current                 Ready                                     1602h   Resource is Current                     
networking  IAMPolicyMember/hub-admin-serviceaccount  Skipped       Unknown                 -                                         -                                               
networking  IAMServiceAccount/hub-fortigatesdn-sa     Skipped       Unknown                 -                                         -                                               
networking  IAMServiceAccount/hub-managementvm-sa     Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-se  Successful    Current                 Ready                                     1602h   Resource is Current                     
policies    ResourceManagerPolicy/compute-require-sh  Successful    Current                 Ready                                     1602h   Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-l  Successful    Current                 Ready                                     1602h   Resource is Current                     
policies    ResourceManagerPolicy/compute-restrict-v  Successful    Current                 Ready                                     1602h   Resource is Current                     
policies    ResourceManagerPolicy/compute-trusted-im  Successful    Current                 Ready                                     1602h   Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-can-ip-  Successful    Current                 Ready                                     1602h   Resource is Current                     
policies    ResourceManagerPolicy/compute-vm-externa  Successful    Current                 Ready                                     1602h   Resource is Current                     
projects    Project/dmu-admin1-hub-kls                Successful    Current                 Ready                                     1602h   Resource is Current                     
projects    Service/dmu-admin1-hub-kls-compute        Successful    Current                 Ready                                     1602h   Resource is Current                     
projects    Service/dmu-admin1-hub-kls-dns            Successful    Current                 Ready                                     1602h   Resource is Current

obriensystems commented 8 months ago

next fix management-project-id not set in member: "serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com" # kpt-set: serviceAccount:networking-sa@${management-project-id}.iam.gserviceaccount.com

but it is set in clz member: "serviceAccount:networking-sa@kcc-kls-cluster3.iam.gserviceaccount.com" # kpt-set: serviceAccount:networking-sa@${management-project-id}.iam.gserviceaccount.com

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions
   Message:               Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest

obriensystems commented 8 months ago

obrien.industries via full script

clean cluster
clean core-landing-zone

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa

 Warning  UpdateFailed  36s (x9 over 6m44s)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).

checking sa permissions for
iam.serviceAccounts.create

wrong project number

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$  gcloud projects list --filter="kcc-oi-629" '--format=value(PROJECT_NUMBER)'
1020702930278

adding single service account admin role to the yakima gke account to test the reconcile

re-kpt

UpdateFailed: 8
UpToDate: 36
Context "gke_kcc-oi-629_northamerica-northeast1_krmapihost-kcc-oi3" modified.
Active namespace is "config-control".
NAME                                                                AGE   READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   25m   False   UpdateFailed   25m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   25m   True    UpToDate   5m28s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          25m   True    UpToDate   5m28s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            25m   True    UpToDate   5m27s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         25m   True    UpToDate   4m21s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           25m   True    UpToDate   4m15s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           25m   True    UpToDate   5m26s

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          25m   True    UpToDate   4m21s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   25m   True    UpToDate   5m3s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   25m   True    UpToDate   4m43s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        25m   True    UpToDate   5m9s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          25m   True    UpToDate   4m43s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 25m   True    UpToDate   4m42s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     25m   True    UpToDate   4m38s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          25m   True    UpToDate   4m54s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       25m   True    UpToDate   4m37s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     25m   True    UpToDate   4m54s
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 25m   True    UpToDate   2m54s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    25m   True    UpToDate   4m54s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 25m   True    UpToDate   4m15s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 25m   True    UpToDate   4m14s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                25m   True    UpToDate   4m14s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   25m   True    UpToDate   4m14s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              25m   True    UpToDate   4m14s

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   25m   True    UpToDate   5m29s
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          25m   True    UpToDate   5m29s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            25m   True    UpToDate   5m28s
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         25m   True    UpToDate   5m28s
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           25m   True    UpToDate   4m15s
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           25m   True    UpToDate   5m27s

NAME                                                                         AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager   25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling           25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager   25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage           25m   True    UpToDate   25m

fixed

next one


iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin

 "permission": "iam.roles.get",

Warning UpdateFailed 1s (x23 over 34m) iamcustomrole-controller Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing organizations/459065442144/roles/gke.firewall.admin: googleapi: Error 403: You don't have permission to get the role at organizations/459065442144/roles/gke.firewall.admin.


## adding organization role administrator

<img width="1852" alt="Screenshot 2023-10-21 at 01 07 55" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/ea5f1012-e536-4bf0-a291-ea1c39ac6fcf">
<img width="878" alt="Screenshot 2023-10-21 at 01 08 35" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/e7071023-c588-4059-b8c6-54d5b3f04b2b">

fixed

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl get gcp NAME AGE READY STATUS STATUS AGE iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin 47m True UpToDate 90s iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin 47m True UpToDate 90s iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin 47m True UpToDate 90s iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin 47m True UpToDate 90s iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin 47m True UpToDate 91s iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin 47m True UpToDate 90s iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin 47m True UpToDate 90s iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin 47m True UpToDate 90s


20231021:1100 - oi

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l true -r false -d false -j false -p kcc-oi-629 existing project: kcc-oi-629 Date: Sat 21 Oct 2023 02:54:58 PM UTC Timestamp: 1697900098 running with: -b kcc-oi -u oi -c false -l true -r false -d false -p kcc-oi-629 Updated property [core/project]. Switched back to boot project kcc-oi Start: 1697900099 unique string: oi REGION: northamerica-northeast1 NETWORK: kcc-ls-vpc SUBNET: kcc-ls-sn CLUSTER: kcc-oi3 Reusing project: kcc-oi-629 CC_PROJECT_ID: kcc-oi-629 BOOT_PROJECT_ID: kcc-oi BILLING_ID: 014479-806359-2F5F85 ORG_ID: 459065442144 Switching to KCC project kcc-oi-629 Updated property [core/project]. Context "gke_kcc-oi-629_northamerica-northeast1_krmapihost-kcc-oi3" modified. Active namespace is "config-control". deploying core-landing-zone get kpt release package solutions/core-landing-zone version 0.3.2 Package "core-landing-zone": Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.3.2 From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit

tag solutions/core-landing-zone/0.3.2 -> FETCH_HEAD Adding package "solutions/core-landing-zone".

Fetched 1 package(s). kpt live init initializing "resourcegroup.yaml" data (namespace: config-control)...success kpt fn render Package "core-landing-zone": [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2" [PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.5s Results: [info] spec.folderRef.external: set field value to "96269513997" [info] metadata.name: set field value to "security-log-bucket-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.projectRef.name: set field value to "logging-project-oi" [info] spec.locked: set field value to "false" [info] spec.retentionDays: set field value to "1" [info] metadata.name: set field value to "platform-and-component-log-bucket-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.projectRef.name: set field value to "logging-project-oi" [info] spec.locked: set field value to "false" [info] spec.retentionDays: set field value to "1" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.resourceRef.name: set field value to "logging-project-oi" [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "logging-project-oi-security-sink" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.resourceRef.name: set field value to "logging-project-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.resourceRef.name: set field value to "logging-project-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.resourceRef.name: set field value to "logging-project-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.resourceRef.name: set field value to "logging-project-oi" [info] metadata.name: set field value to "logging-project-oi" [info] spec.name: set field value to "logging-project-oi" [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85" [info] spec.folderRef.external: set field value to "96269513997" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi" [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi" [info] spec.folderRef.external: set field value to "96269513997" [info] metadata.name: set field value to "dns-project-oi-standard-core-public-dns" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi" [info] spec.dnsName: set field value to "obrien.industries." [info] metadata.name: set field value to "dns-project-oi" [info] spec.name: set field value to "dns-project-oi" [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85" [info] metadata.name: set field value to "dns-project-oi-dns" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi" [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi" [info] spec.folderRef.external: set field value to "96269513997" [info] spec.projectRef.external: set field value to "kcc-oi-629" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi" [info] spec.projectRef.external: set field value to "kcc-oi-629" [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi" [info] spec.projectRef.external: set field value to "kcc-oi-629" [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/_Default" [info] metadata.name: set field value to "kcc-oi-629-cloudbilling" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.name: set field value to "kcc-oi-629-cloudresourcemanager" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.name: set field value to "kcc-oi-629-serviceusage" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.name: set field value to "kcc-oi-629-accesscontextmanager" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "kcc-oi-629" [info] spec.member: set field value to "serviceAccount:gatekeeper-admin-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[gatekeeper-system/gatekeeper-admin]" [info] spec.googleServiceAccount: set field value to "gatekeeper-admin-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:hierarchy-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]" [info] spec.googleServiceAccount: set field value to "hierarchy-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "459065442144" [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]" [info] spec.googleServiceAccount: set field value to "logging-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "459065442144" [info] spec.member: set field value to "serviceAccount:service-1020702930278@gcp-sa-yakima.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "kcc-oi-629" [info] spec.member: set field value to "serviceAccount:service-1020702930278@gcp-sa-yakima.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "kcc-oi-629" [info] spec.member: set field value to "serviceAccount:service-1020702930278@gcp-sa-yakima.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "459065442144" [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "459065442144" [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]" [info] spec.googleServiceAccount: set field value to "networking-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "459065442144" [info] spec.member: set field value to "serviceAccount:policies-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]" [info] spec.googleServiceAccount: set field value to "policies-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "96269513997" [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.resourceRef.external: set field value to "459065442144" [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629" [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]" [info] spec.googleServiceAccount: set field value to "projects-sa@kcc-oi-629.iam.gserviceaccount.com" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] metadata.namespace: set field value to "config-control" [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.listPolicy.allow.values: set field value to "- \"under:organizations/459065442144\"\n" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.listPolicy.allow.values: set field value to "- \"projects/cos-cloud\"\n" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.listPolicy.allow.values: set field value to "- \"@obrien.industries\"\n" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.listPolicy.allow.values: set field value to "- \"C03kdhrkc\"\n" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.organizationRef.external: set field value to "459065442144" [info] metadata.name: set field value to "logging-project-oi-security-sink" [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket-oi" [info] spec.organizationRef.external: set field value to "459065442144" [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/security-log-bucket-oi"

Successfully executed 1 function(s) in 1 package(s). kpt live apply installing inventory ResourceGroup CRD. inventory update started inventory update finished apply phase started namespace/hierarchy apply successful namespace/logging apply successful namespace/networking apply successful namespace/policies apply successful namespace/projects apply successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager apply successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling apply successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager apply successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage apply successful apply phase finished reconcile phase started namespace/hierarchy reconcile successful namespace/logging reconcile successful namespace/networking reconcile successful namespace/policies reconcile successful namespace/projects reconcile successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile successful iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile pending iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile successful iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile successful iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile successful iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile successful iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager reconcile pending service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling reconcile pending service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager reconcile pending service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage reconcile pending iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile failed iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile failed iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager reconcile successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling reconcile successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage reconcile successful service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager reconcile successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile failed iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile successful

iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile successful iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile successful iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile successful iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile successful reconcile phase finished apply phase started rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging apply successful rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control apply successful rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies apply successful rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects apply successful rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects apply successful rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging apply successful rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking apply successful rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies apply successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful folder.resourcemanager.cnrm.cloud.google.com/audits apply successful folder.resourcemanager.cnrm.cloud.google.com/clients apply successful folder.resourcemanager.cnrm.cloud.google.com/services apply successful folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure apply successful project.resourcemanager.cnrm.cloud.google.com/logging-project-oi apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention apply successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access apply successful apply phase finished reconcile phase started rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile successful rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control reconcile successful rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies reconcile successful rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects reconcile successful rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects reconcile successful rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging reconcile successful rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking reconcile successful rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies reconcile successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending project.resourcemanager.cnrm.cloud.google.com/logging-project-oi reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects reconcile failed resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access reconcile failed folder.resourcemanager.cnrm.cloud.google.com/clients reconcile successful folder.resourcemanager.cnrm.cloud.google.com/services reconcile successful folder.resourcemanager.cnrm.cloud.google.com/audits reconcile successful folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects reconcile pending resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access reconcile successful resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects reconcile successful project.resourcemanager.cnrm.cloud.google.com/logging-project-oi reconcile successful reconcile phase finished apply phase started iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config apply successful iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions apply successful iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions apply successful iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions apply successful iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions apply successful logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi apply successful logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi apply successful logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket apply successful project.resourcemanager.cnrm.cloud.google.com/dns-project-oi apply successful apply phase finished reconcile phase started iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile pending iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions reconcile pending iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions reconcile pending iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions reconcile pending iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions reconcile pending logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi reconcile pending logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi reconcile pending logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile pending project.resourcemanager.cnrm.cloud.google.com/dns-project-oi reconcile pending logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi reconcile successful logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi reconcile successful logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile successful iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile successful


<img width="927" alt="Screenshot 2023-10-21 at 11 02 41" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/16c4351a-d185-47f1-8400-d121bf0605de">

issue was main - 0.3.2 is working for clz package

obriensystems commented 8 months ago

Added fix for missing yakima roles https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/578

Checking folders in clz https://cloud.google.com/config-connector/docs/reference/resource-docs/resourcemanager/folder

see

kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true

getting status via https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe folder audits
Error from server (NotFound): folders.resourcemanager.cnrm.cloud.google.com "audits" not found

working for known up services
ichael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe IAMServiceAccount logging-sa
Name:         logging-sa

rerun
onfig-con  IAMServiceAccount/networking-sa           Skipped       Current                 Ready                                     2h      Resource is Current                     
config-con  IAMServiceAccount/policies-sa             Skipped       Current                 Ready                                     2h      Resource is Current                     
config-con  IAMServiceAccount/projects-sa             Skipped       Current                 Ready                                     2h      Resource is Current                     
config-con  Service/kcc-oi-629-accesscontextmanager   Skipped       Current                 Ready                                     2h      Resource is Current                     
config-con  Service/kcc-oi-629-cloudbilling           Skipped       Current                 Ready                                     2h      Resource is Current                     
config-con  Service/kcc-oi-629-cloudresourcemanager   Skipped       Current                 Ready                                     2h      Resource is Current                     
config-con  Service/kcc-oi-629-serviceusage           Skipped       Current                 Ready                                     2h      Resource is Current                     
gatekeeper  ConfigConnectorContext/configconnectorco  Skipped       Current                 <None>                                    2h      status.healthy is true                  
hierarchy   ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-folders-resource-refer  Skipped       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Skipped       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Skipped       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Skipped       Unknown                 -                                         -                                               
hierarchy   Folder/audits                             Skipped       Unknown                 -                                         -                                               
hierarchy   Folder/clients                            Skipped       Unknown                 -                                         -                                               
hierarchy   Folder/services                           Skipped       Unknown                 -                                         -                                               
hierarchy   Folder/services-infrastructure            Skipped       Unknown                 -                                         -                                               
logging     ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogBucket/platform-and-component-  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogBucket/security-log-bucket-oi   Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/logging-project-oi-securi  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-disa  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/mgmt-project-cluster-plat  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -                                               
logging     RoleBinding/allow-logging-resource-refer  Skipped       Unknown                 -                                         -                                               
networking  ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -                                               
networking  DNSManagedZone/dns-project-oi-standard-c  Skipped       Unknown                 -                                         -                                               
policies    ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-gu  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-ne  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-se  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-disable-vp  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-require-os  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-require-sh  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-require-sh  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-restrict-l  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-restrict-s  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-restrict-v  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-skip-defau  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-trusted-im  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-vm-can-ip-  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/compute-vm-externa  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/essentialcontacts-  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/gcp-restrict-resou  Skipped       Unknown                 -                                         -                                               
policies    ResourceManagerPolicy/iam-allowed-policy  Skipped       Unknown                 -                                         -

update folder id (was older one) lz-folder-id: '871256537757'

deleted lz, reran kpt

GKE cluster crashed https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/580

projects    Project/logging-project-oi                Skipped       Unknown                 -                                         -                                               
projects    Service/dns-project-oi-dns                Skipped       Unknown                 -                                         -                                               
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1821bb2]

goroutine 757 [running]:
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GetResourceVersion(...)
        /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.9/pkg/apis/meta/v1/unstructured/unstructured.go:282
github.com/GoogleContainerTools/kpt/pkg/live.(*InventoryResourceGroup).ApplyWithPrune(0xc001692ea0?, {0x221cea0?, 0xc002758530?}, {0x223fd10?, 0xc001fa02a0?}, 0x1, {0x68?, 0x92?, 0x1ca6d60?})
        /home/runner/work/kpt/kpt/pkg/live/inventoryrg.go:299 +0x112
sigs.k8s.io/cli-utils/pkg/inventory.(*ClusterClient).Replace(0xc00118c980, {0x22398e8, 0xc000e89040}, {0xc00072b800?, 0x2c, 0x2c}, {0xc000e16000, 0x68, 0x92}, 0x0)
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/inventory/inventory-client.go:202 +0x830
sigs.k8s.io/cli-utils/pkg/apply/task.(*DeleteOrUpdateInvTask).updateInventory(0xc000cba300, 0xc000dd8400)
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/apply/task/inv_set_task.go:161 +0x294a
sigs.k8s.io/cli-utils/pkg/apply/task.(*DeleteOrUpdateInvTask).Start.func1()
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/apply/task/inv_set_task.go:54 +0x3f
created by sigs.k8s.io/cli-utils/pkg/apply/task.(*DeleteOrUpdateInvTask).Start in goroutine 181
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/apply/task/inv_set_task.go:49 +0x67

recycling cluster

bring up a new cluster in place
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n false -c true -l false -r false -d false -j false -p kcc-oi-629

delete old kpt package - let the script fetch
deploy lz
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l true -r false -d false -j false -p kcc-oi-629

[info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi"
    [info] spec.projectRef.external: set field value to "kcc-oi-629"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/_Default"
    [info] metadata.name: set field value to "kcc-oi-629-cloudbilling"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.name: set field value to "kcc-oi-629-cloudresourcemanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.name: set field value to "kcc-oi-629-serviceusage"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.name: set field value to "kcc-oi-629-accesscontextmanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "kcc-oi-629"
    [info] spec.member: set field value to "serviceAccount:gatekeeper-admin-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[gatekeeper-system/gatekeeper-admin]"
    [info] spec.googleServiceAccount: set field value to "gatekeeper-admin-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:hierarchy-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]"
    [info] spec.googleServiceAccount: set field value to "hierarchy-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]"
    [info] spec.googleServiceAccount: set field value to "logging-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:service-1020702930278@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "kcc-oi-629"
    [info] spec.member: set field value to "serviceAccount:service-1020702930278@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "kcc-oi-629"
    [info] spec.member: set field value to "serviceAccount:service-1020702930278@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]"
    [info] spec.googleServiceAccount: set field value to "networking-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:policies-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]"
    [info] spec.googleServiceAccount: set field value to "policies-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "96269513997"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-629"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-629.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]"
    [info] spec.googleServiceAccount: set field value to "projects-sa@kcc-oi-629.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"under:organizations/459065442144\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/cos-cloud\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"@obrien.industries\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.listPolicy.allow.values: set field value to "- \"C03kdhrkc\"\n"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] metadata.name: set field value to "logging-project-oi-security-sink"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket-oi"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/security-log-bucket-oi"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
NAMESPACE   RESOURCE                                  ACTION        STATUS      RECONCILED  CONDITIONS                                AGE     MESSAGE                                 
            Namespace/hierarchy                       Pending       Unknown                 -                                         -                                               
            Namespace/logging                         Pending       Unknown                 -                                         -                                               
            Namespace/networking                      Pending       Unknown                 -                                         -                                               
            Namespace/policies                        Pending       Unknown                 -                                         -                                               
            Namespace/projects                        Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/gke-firewall-admin          Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier2-dnsrecord-admin       Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier2-vpcpeering-admin      Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier3-dnsrecord-admin       Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier3-firewallrule-admin    Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier3-subnetwork-admin      Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier3-vpcsc-admin           Pending       Unknown                 -                                         -                                               
config-con  IAMCustomRole/tier4-secretmanager-admin   Pending       Unknown                 -                                         -                                               
config-con  IAMPartialPolicy/gatekeeper-admin-sa-wor  Pending       Unknown                 -                                         -                                               
config-con  IAMPartialPolicy/hierarchy-sa-workload-i  Pending       Unknown                 -                                         -                                               
config-con  IAMPartialPolicy/logging-sa-workload-ide  Pending       Unknown                 -                                         -                                               
config-con  IAMPartialPolicy/networking-sa-workload-  Pending       Unknown                 -                                         -                                               
config-con  IAMPartialPolicy/policies-sa-workload-id  Pending       Unknown                 -                                         -                                               
config-con  IAMPartialPolicy/projects-sa-workload-id  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/config-control-sa-manage  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/config-control-sa-manage  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/config-control-sa-orgrol  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/gatekeeper-admin-sa-metr  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/hierarchy-sa-folderadmin  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/logging-sa-bigqueryadmin  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/logging-sa-logadmin-perm  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-dns-permis  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-networkadm  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-security-p  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-service-co  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-servicedir  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/networking-sa-xpnadmin-p  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/policies-sa-orgpolicyadm  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/projects-sa-billinguser-  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/projects-sa-projectcreat  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/projects-sa-projectdelet  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/projects-sa-projectiamad  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/projects-sa-projectmover  Pending       Unknown                 -                                         -                                               
config-con  IAMPolicyMember/projects-sa-serviceusage  Pending       Unknown                 -                                         -                                               
config-con  IAMServiceAccount/gatekeeper-admin-sa     Pending       Unknown                 -                                         -                                               
config-con  IAMServiceAccount/hierarchy-sa            Pending       Unknown                 -                                         -                                               
config-con  IAMServiceAccount/logging-sa              Pending       Unknown                 -                                         -                                               
config-con  IAMServiceAccount/networking-sa           Pending       Unknown                 -                                         -                                               
config-con  IAMServiceAccount/policies-sa             Pending       Unknown                 -                                         -                                               
config-con  IAMServiceAccount/projects-sa             Pending       Unknown                 -                                         -                                               
config-con  Service/kcc-oi-629-accesscontextmanager   Pending       Unknown                 -                                         -                                               
config-con  Service/kcc-oi-629-cloudbilling           Pending       Unknown                 -                                         -                                               
config-con  Service/kcc-oi-629-cloudresourcemanager   Pending       Unknown                 -                                         -                                               
config-con  Service/kcc-oi-629-serviceusage           Pending       Unknown                 -                                         -                                               
gatekeeper  ConfigConnectorContext/configconnectorco  Pending       Unknown                 -                                         -                                               
hierarchy   ConfigConnectorContext/configconnectorco  Pending       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-folders-resource-refer  Pending       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Pending       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Pending       Unknown                 -                                         -                                               
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Pending       Unknown                 -                                         -                                               
hierarchy   Folder/audits                             Pending       Unknown                 -                                         -                                               
hierarchy   Folder/clients                            Pending       Unknown                 -                                         -                                               
hierarchy   Folder/services                           Pending       Unknown                 -                                         -                                               
hierarchy   Folder/services-infrastru

check the kpt alpha live plan

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kpt alpha live plan core-landing-zone
error: 12 errors:
- invalid object: "logging_security-log-bucket_logging.cnrm.cloud.google.com_LoggingLogBucket": invalid "config.kubernetes.io/depends-on" annotation: external dependency: logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "logging_platform-and-component-log-bucket_logging.cnrm.cloud.google.com_LoggingLogBucket": invalid "config.kubernetes.io/depends-on" annotation: external dependency: logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "projects_security-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/security-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "projects_platform-and-component-services-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/platform-and-component-services-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "projects_platform-and-component-services-infra-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/platform-and-component-services-infra-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "projects_mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "projects_logging-project-data-access-log-config_iam.cnrm.cloud.google.com_IAMAuditConfig": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMAuditConfig/logging-project-data-access-log-config -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "logging_platform-and-component-services-log-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "platform-and-component-log-bucket"
- invalid object: "logging_platform-and-component-services-infra-log-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "platform-and-component-log-bucket"
- invalid object: "logging_mgmt-project-cluster-platform-and-component-log-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "platform-and-component-log-bucket"
- invalid object: "logging_mgmt-project-cluster-disable-default-bucket_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: external dependency: logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogSink/mgmt-project-cluster-disable-default-bucket -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
- invalid object: "logging_logging-project-id-security-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "security-log-bucket"

obriensystems commented 8 months ago

CC get resource status https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources
CFT https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/config-connector/solutions
KCC https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions
Simple PubSub example to get familiar with https://cloud.google.com/config-connector/docs/how-to/getting-started
KPT reference https://kpt.dev/reference/schema/plan/
Google Architecture Docs on config controller and config sync https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt

obriensystems commented 8 months ago

LZ inventory - kls - folders ok

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get namespaces
NAME                              STATUS   AGE
cnrm-system                       Active   79d
config-control                    Active   79d
config-management-monitoring      Active   79d
config-management-system          Active   79d
configconnector-operator-system   Active   79d
default                           Active   79d
gatekeeper-system                 Active   79d
gke-gmp-system                    Active   79d
gke-managed-filestorecsi          Active   79d
gmp-public                        Active   79d
hierarchy                         Active   67d
krmapihosting-monitoring          Active   79d
krmapihosting-system              Active   79d
kube-node-lease                   Active   79d
kube-public                       Active   79d
kube-system                       Active   79d
logging                           Active   67d
networking                        Active   67d
policies                          Active   67d
projects                          Active   67d
resource-group-system             Active   79d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n logging
NAME                                                                                   AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-kls   67d   True    UpToDate   67d
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-kls                 67d   True    UpToDate   67d

NAME                                                                                                AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/logging-project-kls-security-sink                      67d   True    UpToDate   67d
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket            67d   True    UpToDate   67d
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-sink   67d   True    UpToDate   67d
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-infra-log-sink         67d   True    UpToDate   67d
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-log-sink               67d   True    UpToDate   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    67d   True    UpToDate   67d
folder.resourcemanager.cnrm.cloud.google.com/clients                   67d   True    UpToDate   67d
folder.resourcemanager.cnrm.cloud.google.com/services                  67d   True    UpToDate   67d
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   67d   True    UpToDate   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking
NAME                                                                             AGE   READY   STATUS     STATUS AGE
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address         67d   True    UpToDate   22h
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address         67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address        67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address     67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address       67d   True    UpToDate   22h
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address       67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address      67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address   67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address                     67d   True    UpToDate   67d
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address               67d   True    UpToDate   67d

NAME                                                                                               AGE   READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                               67d   True    UpToDate             11d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d

NAME                                                                       AGE   READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d

NAME                                                                   AGE   READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   67d   True    UpToDate   5m25s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   67d   True    UpToDate   7m52s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       67d   True    UpToDate   5s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    67d   True    UpToDate   2m46s

NAME                                                                    AGE   READY   STATUS     STATUS AGE
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat   67d   True    UpToDate   67d

NAME                                                                    AGE   READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   67d   True    UpToDate   6d23h

NAME                                                                                AGE   READY   STATUS     STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   67d   True    UpToDate   2d15h

NAME                                                                          AGE   READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   67d   True    UpToDate   22h
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   67d   True    UpToDate   3h53m
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        67d   True    UpToDate   2d15h
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    67d   True    UpToDate   32d

NAME                                                                                AGE   READY   STATUS     STATUS AGE
dnsmanagedzone.dns.cnrm.cloud.google.com/dns-project-kls-standard-core-public-dns   67d   True    UpToDate   67d

NAME                                                                 AGE   READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   67d   True    UpToDate   22h
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   67d   True    UpToDate   22h
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       67d   True    UpToDate   9h
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    67d   True    UpToDate   22h

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   67d   True    UpToDate   67d

NAME                                                                                                                   AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   67d   True    UpToDate   67d
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         67d   True    UpToDate   67d
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               67d   True    UpToDate   67d
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      67d   True    UpToDate   67d

NAME                                                                AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dmu-admin1-hub-kls    67d   True    UpToDate   67d
project.resourcemanager.cnrm.cloud.google.com/dns-project-kls       67d   True    UpToDate   67d
project.resourcemanager.cnrm.cloud.google.com/logging-project-kls   67d   True    UpToDate   11h

NAME                                                                    AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/dmu-admin1-hub-kls-compute   67d   True    UpToDate   67d
service.serviceusage.cnrm.cloud.google.com/dmu-admin1-hub-kls-dns       67d   True    UpToDate   67d
service.serviceusage.cnrm.cloud.google.com/dns-project-kls-dns          67d   True    UpToDate   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies
NAME                                                                                                                               AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access                                 67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization                                  67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access                                     67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project                  67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6                                      67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login                                               67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm                                            67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project                         67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project                         67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types                      67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project   67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal                               67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering                                           67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project                        67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation                                  67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects                                         67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project                      67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward                                              67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project                           67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access                                          67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project                       67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains                              67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations                                        67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains                                      67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation                               67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip                                                 67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention                                       67d   True    UpToDate   67d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access                                    67d   True    UpToDate   67d

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg tree core-landing-zone
Package "core-landing-zone"
├── [Kptfile]  Kptfile core-landing-zone
├── [resourcegroup.yaml]  ResourceGroup config-control/inventory-85852139
├── [setters.yaml]  ConfigMap setters
├── audits
│   ├── [folder.yaml]  Folder hierarchy/audits
│   └── logging-project
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/platform-and-component-log-bucket-kls
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/security-log-bucket-kls
│       ├── [project-iam.yaml]  IAMAuditConfig projects/logging-project-data-access-log-config
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-infra-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/security-log-bucket-writer-permissions
│       └── [project.yaml]  Project projects/logging-project-kls
├── clients
│   └── [folder.yaml]  Folder hierarchy/clients
├── services
│   ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-log-sink
│   ├── [folder.yaml]  Folder hierarchy/services
│   └── services-infrastructure
│       ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-infra-log-sink
│       ├── [folder.yaml]  Folder hierarchy/services-infrastructure
│       └── dns-project
│           ├── [dns.yaml]  DNSManagedZone networking/dns-project-kls-standard-core-public-dns
│           ├── [project.yaml]  Project projects/dns-project-kls
│           └── [services.yaml]  Service projects/dns-project-kls-dns
├── mgmt-project
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-disable-default-bucket
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink
│   ├── [services.yaml]  Service config-control/kcc-kls-cluster3-accesscontextmanager
│   ├── [services.yaml]  Service config-control/kcc-kls-cluster3-cloudbilling
│   ├── [services.yaml]  Service config-control/kcc-kls-cluster3-cloudresourcemanager
│   ├── [services.yaml]  Service config-control/kcc-kls-cluster3-serviceusage
│   └── org-policies
│       └── [compute-require-shielded-vm-except-mgmt-project.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm-except-mgt-project
├── namespaces
│   ├── [gatekeeper-system.yaml]  IAMServiceAccount config-control/gatekeeper-admin-sa
│   ├── [gatekeeper-system.yaml]  IAMPolicyMember config-control/gatekeeper-admin-sa-metric-writer-permissions
│   ├── [gatekeeper-system.yaml]  IAMPartialPolicy config-control/gatekeeper-admin-sa-workload-identity-binding
│   ├── [gatekeeper-system.yaml]  ConfigConnectorContext gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [hierarchy.yaml]  Namespace hierarchy
│   ├── [hierarchy.yaml]  IAMServiceAccount config-control/hierarchy-sa
│   ├── [hierarchy.yaml]  IAMPolicyMember config-control/hierarchy-sa-folderadmin-permissions
│   ├── [hierarchy.yaml]  IAMPartialPolicy config-control/hierarchy-sa-workload-identity-binding
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-folders-resource-reference-to-logging
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-config-control
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-policies
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-projects
│   ├── [hierarchy.yaml]  ConfigConnectorContext hierarchy/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  Namespace logging
│   ├── [logging.yaml]  IAMServiceAccount config-control/logging-sa
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-bigqueryadmin-permissions
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-logadmin-permissions
│   ├── [logging.yaml]  IAMPartialPolicy config-control/logging-sa-workload-identity-binding
│   ├── [logging.yaml]  RoleBinding logging/allow-logging-resource-reference-from-projects
│   ├── [logging.yaml]  ConfigConnectorContext logging/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-editor-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-serviceaccountadmin-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-orgroleadmin-permissions
│   ├── [networking.yaml]  Namespace networking
│   ├── [networking.yaml]  IAMServiceAccount config-control/networking-sa
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-dns-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-networkadmin-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-security-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-service-control-org-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-servicedirectoryeditor-permissions
│   ├── [networking.yaml]  IAMPartialPolicy config-control/networking-sa-workload-identity-binding
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-xpnadmin-permissions
│   ├── [networking.yaml]  ConfigConnectorContext networking/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [policies.yaml]  Namespace policies
│   ├── [policies.yaml]  IAMServiceAccount config-control/policies-sa
│   ├── [policies.yaml]  IAMPolicyMember config-control/policies-sa-orgpolicyadmin-permissions
│   ├── [policies.yaml]  IAMPartialPolicy config-control/policies-sa-workload-identity-binding
│   ├── [policies.yaml]  ConfigConnectorContext policies/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [projects.yaml]  Namespace projects
│   ├── [projects.yaml]  IAMServiceAccount config-control/projects-sa
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-billinguser-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectcreator-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectdeleter-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectiamadmin-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectmover-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-serviceusageadmin-permissions
│   ├── [projects.yaml]  IAMPartialPolicy config-control/projects-sa-workload-identity-binding
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-logging
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-networking
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-policies
│   └── [projects.yaml]  ConfigConnectorContext projects/configconnectorcontext.core.cnrm.cloud.google.com
└── org
    ├── [org-sink.yaml]  LoggingLogSink logging/logging-project-kls-security-sink
    ├── custom-roles
    │   ├── [gke-firewall-admin.yaml]  IAMCustomRole config-control/gke-firewall-admin
    │   ├── [tier2-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier2-dnsrecord-admin
    │   ├── [tier2-vpcpeering-admin.yaml]  IAMCustomRole config-control/tier2-vpcpeering-admin
    │   ├── [tier3-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier3-dnsrecord-admin
    │   ├── [tier3-firewallrule-admin.yaml]  IAMCustomRole config-control/tier3-firewallrule-admin
    │   └── [tier3-vpcsc-admin.yaml]  IAMCustomRole config-control/tier3-vpcsc-admin
    └── org-policies
        ├── [compute-disable-guest-attribute-access.yaml]  ResourceManagerPolicy policies/compute-disable-guest-attribute-access
        ├── [compute-disable-nested-virtualization.yaml]  ResourceManagerPolicy policies/compute-disable-nested-virtualization
        ├── [compute-disable-serial-port-access.yaml]  ResourceManagerPolicy policies/compute-disable-serial-port-access
        ├── [compute-disable-vpc-external-ipv6.yaml]  ResourceManagerPolicy policies/compute-disable-vpc-external-ipv6
        ├── [compute-require-os-login.yaml]  ResourceManagerPolicy policies/compute-require-os-login
        ├── [compute-require-shielded-vm.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm
        ├── [compute-restrict-load-balancer-creation-for-types.yaml]  ResourceManagerPolicy policies/compute-restrict-load-balancer-creation-for-types
        ├── [compute-restrict-shared-vpc-lien-removal.yaml]  ResourceManagerPolicy policies/compute-restrict-shared-vpc-lien-removal
        ├── [compute-restrict-vpc-peering.yaml]  ResourceManagerPolicy policies/compute-restrict-vpc-peering
        ├── [compute-skip-default-network-creation.yaml]  ResourceManagerPolicy policies/compute-skip-default-network-creation
        ├── [compute-trusted-image-projects.yaml]  ResourceManagerPolicy policies/compute-trusted-image-projects
        ├── [compute-vm-can-ip-forward.yaml]  ResourceManagerPolicy policies/compute-vm-can-ip-forward
        ├── [compute-vm-external-ip-access.yaml]  ResourceManagerPolicy policies/compute-vm-external-ip-access
        ├── [essentialcontacts-allowed-contact-domains.yaml]  ResourceManagerPolicy policies/essentialcontacts-allowed-contact-domains
        ├── [gcp-resource-locations.yaml]  ResourceManagerPolicy policies/gcp-restrict-resource-locations
        ├── [iam-allowed-policy-member-domains.yaml]  ResourceManagerPolicy policies/iam-allowed-policy-member-domains
        ├── [iam-disable-service-account-key-creation.yaml]  ResourceManagerPolicy policies/iam-disable-service-account-key-creation
        ├── [sql-restrict-public-ip.yaml]  ResourceManagerPolicy policies/sql-restrict-public-ip
        ├── [storage-public-access-prevention.yaml]  ResourceManagerPolicy policies/storage-public-access-prevention
        └── [storage-uniform-bucket-level-access.yaml]  ResourceManagerPolicy policies/storage-uniform-bucket-level-access

LZ inventory - oi - no folders

michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt pkg tree core-landing-zone
Package "core-landing-zone"
├── [Kptfile]  Kptfile core-landing-zone
├── [resourcegroup.yaml]  ResourceGroup config-control/inventory-49021548
├── [setters.yaml]  ConfigMap setters
├── audits
│   ├── [folder.yaml]  Folder hierarchy/audits
│   └── logging-project
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/platform-and-component-log-bucket-oi
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/security-log-bucket-oi
│       ├── [project-iam.yaml]  IAMAuditConfig projects/logging-project-data-access-log-config
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-infra-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/security-log-bucket-writer-permissions
│       └── [project.yaml]  Project projects/logging-project-oi
├── clients
│   └── [folder.yaml]  Folder hierarchy/clients
├── services
│   ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-log-sink
│   ├── [folder.yaml]  Folder hierarchy/services
│   └── services-infrastructure
│       ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-infra-log-sink
│       ├── [folder.yaml]  Folder hierarchy/services-infrastructure
│       └── dns-project
│           ├── [dns.yaml]  DNSManagedZone networking/dns-project-oi-standard-core-public-dns
│           ├── [project.yaml]  Project projects/dns-project-oi
│           └── [services.yaml]  Service projects/dns-project-oi-dns
├── mgmt-project
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-disable-default-bucket
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink
│   ├── [services.yaml]  Service config-control/kcc-oi-629-accesscontextmanager
│   ├── [services.yaml]  Service config-control/kcc-oi-629-cloudbilling
│   ├── [services.yaml]  Service config-control/kcc-oi-629-cloudresourcemanager
│   ├── [services.yaml]  Service config-control/kcc-oi-629-serviceusage
│   └── org-policies
│       └── [compute-require-shielded-vm-except-mgmt-project.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm-except-mgt-project
├── namespaces
│   ├── [gatekeeper-system.yaml]  IAMServiceAccount config-control/gatekeeper-admin-sa
│   ├── [gatekeeper-system.yaml]  IAMPolicyMember config-control/gatekeeper-admin-sa-metric-writer-permissions
│   ├── [gatekeeper-system.yaml]  IAMPartialPolicy config-control/gatekeeper-admin-sa-workload-identity-binding
│   ├── [gatekeeper-system.yaml]  ConfigConnectorContext gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [hierarchy.yaml]  Namespace hierarchy
│   ├── [hierarchy.yaml]  IAMServiceAccount config-control/hierarchy-sa
│   ├── [hierarchy.yaml]  IAMPolicyMember config-control/hierarchy-sa-folderadmin-permissions
│   ├── [hierarchy.yaml]  IAMPartialPolicy config-control/hierarchy-sa-workload-identity-binding
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-folders-resource-reference-to-logging
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-config-control
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-policies
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-projects
│   ├── [hierarchy.yaml]  ConfigConnectorContext hierarchy/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  Namespace logging
│   ├── [logging.yaml]  IAMServiceAccount config-control/logging-sa
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-bigqueryadmin-permissions
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-logadmin-permissions
│   ├── [logging.yaml]  IAMPartialPolicy config-control/logging-sa-workload-identity-binding
│   ├── [logging.yaml]  RoleBinding logging/allow-logging-resource-reference-from-projects
│   ├── [logging.yaml]  ConfigConnectorContext logging/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-editor-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-serviceaccountadmin-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-orgroleadmin-permissions
│   ├── [networking.yaml]  Namespace networking
│   ├── [networking.yaml]  IAMServiceAccount config-control/networking-sa
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-dns-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-networkadmin-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-security-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-service-control-org-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-servicedirectoryeditor-permissions
│   ├── [networking.yaml]  IAMPartialPolicy config-control/networking-sa-workload-identity-binding
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-xpnadmin-permissions
│   ├── [networking.yaml]  ConfigConnectorContext networking/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [policies.yaml]  Namespace policies
│   ├── [policies.yaml]  IAMServiceAccount config-control/policies-sa
│   ├── [policies.yaml]  IAMPolicyMember config-control/policies-sa-orgpolicyadmin-permissions
│   ├── [policies.yaml]  IAMPartialPolicy config-control/policies-sa-workload-identity-binding
│   ├── [policies.yaml]  ConfigConnectorContext policies/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [projects.yaml]  Namespace projects
│   ├── [projects.yaml]  IAMServiceAccount config-control/projects-sa
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-billinguser-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectcreator-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectdeleter-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectiamadmin-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectmover-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-serviceusageadmin-permissions
│   ├── [projects.yaml]  IAMPartialPolicy config-control/projects-sa-workload-identity-binding
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-logging
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-networking
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-policies
│   └── [projects.yaml]  ConfigConnectorContext projects/configconnectorcontext.core.cnrm.cloud.google.com
└── org
    ├── [org-sink.yaml]  LoggingLogSink logging/logging-project-oi-security-sink
    ├── custom-roles
    │   ├── [gke-firewall-admin.yaml]  IAMCustomRole config-control/gke-firewall-admin
    │   ├── [tier2-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier2-dnsrecord-admin
    │   ├── [tier2-vpcpeering-admin.yaml]  IAMCustomRole config-control/tier2-vpcpeering-admin
    │   ├── [tier3-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier3-dnsrecord-admin
    │   ├── [tier3-firewallrule-admin.yaml]  IAMCustomRole config-control/tier3-firewallrule-admin
    │   ├── [tier3-subnetwork-admin.yaml]  IAMCustomRole config-control/tier3-subnetwork-admin
    │   ├── [tier3-vpcsc-admin.yaml]  IAMCustomRole config-control/tier3-vpcsc-admin
    │   └── [tier4-secretmanager-admin.yaml]  IAMCustomRole config-control/tier4-secretmanager-admin
    └── org-policies
        ├── [compute-disable-guest-attribute-access.yaml]  ResourceManagerPolicy policies/compute-disable-guest-attribute-access
        ├── [compute-disable-nested-virtualization.yaml]  ResourceManagerPolicy policies/compute-disable-nested-virtualization
        ├── [compute-disable-serial-port-access.yaml]  ResourceManagerPolicy policies/compute-disable-serial-port-access
        ├── [compute-disable-vpc-external-ipv6.yaml]  ResourceManagerPolicy policies/compute-disable-vpc-external-ipv6
        ├── [compute-require-os-login.yaml]  ResourceManagerPolicy policies/compute-require-os-login
        ├── [compute-require-shielded-vm.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm
        ├── [compute-restrict-load-balancer-creation-for-types.yaml]  ResourceManagerPolicy policies/compute-restrict-load-balancer-creation-for-types
        ├── [compute-restrict-shared-vpc-lien-removal.yaml]  ResourceManagerPolicy policies/compute-restrict-shared-vpc-lien-removal
        ├── [compute-restrict-vpc-peering.yaml]  ResourceManagerPolicy policies/compute-restrict-vpc-peering
        ├── [compute-skip-default-network-creation.yaml]  ResourceManagerPolicy policies/compute-skip-default-network-creation
        ├── [compute-trusted-image-projects.yaml]  ResourceManagerPolicy policies/compute-trusted-image-projects
        ├── [compute-vm-can-ip-forward.yaml]  ResourceManagerPolicy policies/compute-vm-can-ip-forward
        ├── [compute-vm-external-ip-access.yaml]  ResourceManagerPolicy policies/compute-vm-external-ip-access
        ├── [essentialcontacts-allowed-contact-domains.yaml]  ResourceManagerPolicy policies/essentialcontacts-allowed-contact-domains
        ├── [gcp-resource-locations.yaml]  ResourceManagerPolicy policies/gcp-restrict-resource-locations
        ├── [iam-allowed-policy-member-domains.yaml]  ResourceManagerPolicy policies/iam-allowed-policy-member-domains
        ├── [iam-disable-service-account-key-creation.yaml]  ResourceManagerPolicy policies/iam-disable-service-account-key-creation
        ├── [sql-restrict-public-ip.yaml]  ResourceManagerPolicy policies/sql-restrict-public-ip
        ├── [storage-public-access-prevention.yaml]  ResourceManagerPolicy policies/storage-public-access-prevention
        └── [storage-uniform-bucket-level-access.yaml]  ResourceManagerPolicy policies/storage-uniform-bucket-level-access

Additions to triage since core-landing-zone was last working

obriensystems commented 8 months ago

Hold: remaining services on

kcc.landing.systems

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies | grep false
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n projects | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n hierarchy | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n logging | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$

obrien.industries

Fix for core-landing-zone (regression on my 2nd clean env) - was to not use main - the 0.3.2 release is working like my older hub-env from 60 days ago

raised another issue on the 12 yaml problems since the last release in 0.3.2 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/584

still in clz- issues with main fixed by going to 0.3.2 solutions/core-landing-zone/0.3.2 -> FETCH_HEAD


michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kpt alpha live plan core-landing-zone
error: 12 errors:

invalid object: "logging_security-log-bucket_logging.cnrm.cloud.google.com_LoggingLogBucket": invalid "config.kubernetes.io/depends-on" annotation: external dependency: logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "logging_platform-and-component-log-bucket_logging.cnrm.cloud.google.com_LoggingLogBucket": invalid "config.kubernetes.io/depends-on" annotation: external dependency: logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "projects_security-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/security-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "projects_platform-and-component-services-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/platform-and-component-services-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "projects_platform-and-component-services-infra-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/platform-and-component-services-infra-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "projects_mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions_iam.cnrm.cloud.google.com_IAMPartialPolicy": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMPartialPolicy/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "projects_logging-project-data-access-log-config_iam.cnrm.cloud.google.com_IAMAuditConfig": invalid "config.kubernetes.io/depends-on" annotation: external dependency: iam.cnrm.cloud.google.com/namespaces/projects/IAMAuditConfig/logging-project-data-access-log-config -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "logging_platform-and-component-services-log-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "platform-and-component-log-bucket"
invalid object: "logging_platform-and-component-services-infra-log-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "platform-and-component-log-bucket"
invalid object: "logging_mgmt-project-cluster-platform-and-component-log-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "platform-and-component-log-bucket"
invalid object: "logging_mgmt-project-cluster-disable-default-bucket_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: external dependency: logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogSink/mgmt-project-cluster-disable-default-bucket -> resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${logging-project-id}
invalid object: "logging_logging-project-id-security-sink_logging.cnrm.cloud.google.com_LoggingLogSink": invalid "config.kubernetes.io/depends-on" annotation: failed to parse object reference (index: 0): expected 3 or 5 fields, found 1: "security-log-bucket"


<img width="927" alt="Screenshot 2023-10-21 at 11 02 41" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/16c4351a-d185-47f1-8400-d121bf0605de">

issue was main - 0.3.2 is working for clz package

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp NAME AGE READY STATUS STATUS AGE iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin 5h4m True UpToDate 5h4m iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin 5h4m True UpToDate 5h4m

NAME AGE READY STATUS STATUS AGE iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding 5h4m True UpToDate 5h4m iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding 5h4m True UpToDate 5h4m iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding 5h4m True UpToDate 5h4m iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding 5h4m True UpToDate 5h4m iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding 5h4m True UpToDate 5h3m iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding 5h4m True UpToDate 5h4m

NAME iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-e iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-s iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions AGE READY STATUS STATUS AGE ditor-permissions 5h4m True UpToDate 5h4m erviceaccountadmin-permissions 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h4m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h3m 5h4m True UpToDate 5h4m

NAME AGE READY STATUS STATUS AGE iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa 5h4m True UpToDate 5h4m iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa 5h4m True UpToDate 5h4m iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa 5h4m True UpToDate 5h4m iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa 5h4m True UpToDate 5h4m iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa 5h4m True UpToDate 5h3m iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa 5h4m True UpToDate 5h4m

NAME AGE READY STATUS STATUS AGE service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager 5h4m True UpToDate 5h4m service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling 5h4m True UpToDate 5h4m service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager 5h4m True UpToDate 5h4m service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage 5h4m True UpToDate 5h4m michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp -n hierarchy NAME AGE READY STATUS STATUS AGE folder.resourcemanager.cnrm.cloud.google.com/audits 5h3m True UpToDate 5h3m folder.resourcemanager.cnrm.cloud.google.com/clients 5h3m True UpToDate 5h3m folder.resourcemanager.cnrm.cloud.google.com/services 5h3m True UpToDate 5h3m folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure 5h3m True UpToDate 5h3m michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp -n policies NAME AGE READY STATUS STATUS AGE resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention 5h4m True UpToDate 5h3m resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access 5h4m True UpToDate 5h3m michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp -n logging NAME AGE READY STATUS STATUS AGE logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi 5h3m True UpToDate 5h3m logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi 5h3m True UpToDate 5h3m

NAME AGE READY STATUS STATUS AGE logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket 5h3m True UpToDate 5h3m michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp -n projects NAME AGE READY STATUS STATUS AGE iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config 5h3m True UpToDate 5h3m

NAME AGE READY STATUS STATUS AGE iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions 5h3m False DependencyNotFound 5h3m iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions 5h3m False DependencyNotFound 5h3m iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions 5h3m False DependencyNotFound 5h3m iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions 5h3m False DependencyNotFound 5h3m

NAME AGE READY STATUS STATUS AGE project.resourcemanager.cnrm.cloud.google.com/dns-project-oi 5h3m True UpToDate 5h2m project.resourcemanager.cnrm.cloud.google.com/logging-project-oi 5h5m True UpToDate 5h3m michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$

one of them is waiting on a sink michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions -n projects Name: security-log-bucket-writer-permissions Status: Conditions: Last Transition Time: 2023-10-21T14:58:11Z Message: reference LoggingLogSink logging/logging-project-oi-security-sink is not found Reason: DependencyNotFound Status: False

the bucket is up for

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl describe logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi -n logging

referenced by

Warning DependencyNotFound 5m37s (x32 over 5h16m) iampartialpolicy-controller reference LoggingLogSink logging/logging-project-oi-security-sink is not found michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions -n projects

Message:               reference LoggingLogSink logging/logging-project-oi-security-sink is not found

loggingLogBucketRef:

destination.loggingLogBucketRef

  # Only `external` field is supported to configure the reference.
  external: logging.googleapis.com/projects/logging-project-oi/locations/northamerica-northeast1/buckets/security-log-bucket-oi # kpt-set: logging.googleapis.com/projects/${logging-project-id}/locations/northamerica-northeast1/buckets/${security-log-bucket}

<img width="1817" alt="Screenshot 2023-10-21 at 16 23 15" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/3391b39c-bbde-4782-b370-53ad6e57e44d">

<img width="1353" alt="Screenshot 2023-10-21 at 16 16 33" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/0b8842f0-4c4a-4525-bdc5-27c7fb7c63b4">

Missing LoggingLogSink - actually no sinks in asset inventory
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/core-landing-zone/org/org-sink.yaml#L18

comparing there is a regression since 60days ago

before root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n projects NAME AGE READY STATUS STATUS AGE iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config 67d True UpToDate 67d

NAME AGE READY STATUS STATUS AGE iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions 67d True UpToDate 67d iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions 67d True UpToDate 67d iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions 67d True UpToDate 67d iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions 67d True UpToDate 67d

NAME AGE READY STATUS STATUS AGE project.resourcemanager.cnrm.cloud.google.com/logging-project-kls 68d True UpToDate 17h

latest 0.3.2 michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp -n projects NAME AGE READY STATUS STATUS AGE iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config 5h32m True UpToDate 5h32m

NAME AGE READY STATUS STATUS AGE iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions 5h32m False DependencyNotFound 5h32m iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions 5h32m False DependencyNotFound 5h32m iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions 5h32m False DependencyNotFound 5h32m iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions 5h32m False DependencyNotFound 5h32m

NAME AGE READY STATUS STATUS AGE project.resourcemanager.cnrm.cloud.google.com/dns-project-oi 5h32m True UpToDate 5h31m project.resourcemanager.cnrm.cloud.google.com/logging-project-oi 5h34m True UpToDate 5h32m

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions -n projects Name: security-log-bucket-writer-permissions Namespace: projects Labels: Annotations: cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live config.k8s.io/owning-inventory: 19949defd55b6056ef347db3476403624fa08e71-1697900104974084255 config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi internal.kpt.dev/upstream-identifier: iam.cnrm.cloud.google.com	IAMPartialPolicy	projects	security-log-bucket-writer-permissions API Version: iam.cnrm.cloud.google.com/v1beta1 Kind: IAMPartialPolicy Metadata: Creation Timestamp: 2023-10-21T14:58:10Z Generation: 1 Resource Version: 61577 UID: b4c6e982-c0b2-48d5-8699-bc62ce249673 Spec: Bindings: Members: Member From: Log Sink Ref: Name: logging-project-oi-security-sink Namespace: logging Role: roles/logging.bucketWriter Resource Ref: API Version: resourcemanager.cnrm.cloud.google.com/v1beta1 Kind: Project Name: logging-project-oi Namespace: projects Status: Conditions: Last Transition Time: 2023-10-21T14:58:11Z Message: reference LoggingLogSink logging/logging-project-oi-security-sink is not found Reason: DependencyNotFound Status: False Type: Ready Observed Generation: 1 Events: Type Reason Age From Message

Warning DependencyNotFound 71s (x35 over 5h35m) iampartialpolicy-controller reference LoggingLogSink logging/logging-project-oi-security-sink is not found

the sink is there though


raised for later https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/586
## landing.systems
- creating

obriensystems commented 8 months ago

Triage a way to managed a full lz delete and redeploy - with cycled project names (will need random ids in the future) Issue here is a previously unmet iam role on the log sink is causing issues with deletion

https://console.cloud.google.com/kubernetes/object/logging.cnrm.cloud.google.com/logginglogsinks/northamerica-northeast1/krmapihost-kcc-oi3/logging/mgmt-project-cluster-disable-default-bucket?apiVersion=v1beta1&project=kcc-oi-629&orgonly=true&supportedpurview=project

see #446

status:
  conditions:
  - lastTransitionTime: "2023-10-21T21:13:45Z"
    message: |
      Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Project Logging Sink _Default: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
      Details:
      [
        {
          "@type": "type.googleapis.com/google.rpc.ErrorInfo",
          "domain": "googleapis.com",
          "metadata": {
            "email": "logging-sa@kcc-oi-629.iam.gserviceaccount.com",
            "method": "google.logging.v2.ConfigServiceV2.GetSink",
            "service": "logging.googleapis.com"
          },
          "reason": "ACCOUNT_STATE_INVALID"
        }
      ]

      More details:
      Reason: authError, Message: Invalid Credentials
    reason: DeleteFailed
    status: "False"
    type: Ready

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ cd ../../../kpt
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kpt live destroy core-landing-zone
delete phase started
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket delete successful
delete phase finished
reconcile phase started
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile pending
^C
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl get gcp
NAME                                                                AGE     READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           6h32m   True    UpToDate   6h32m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   6h32m   True    UpToDate   6h32m

NAME                                                                                       AGE     READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   6h32m   True    UpToDate   6h32m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          6h32m   True    UpToDate   6h32m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            6h32m   True    UpToDate   6h32m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         6h32m   True    UpToDate   6h32m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           6h32m   True    UpToDate   6h31m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           6h32m   True    UpToDate   6h32m

NAME                                                                                                             AGE     READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 6h32m   True    UpToDate   6h30m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    6h32m   True    UpToDate   6h32m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   6h32m   True    UpToDate   6h31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              6h32m   True    UpToDate   6h32m

NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   6h32m   True    UpToDate   6h32m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          6h32m   True    UpToDate   6h32m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            6h32m   True    UpToDate   6h32m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         6h32m   True    UpToDate   6h32m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           6h32m   True    UpToDate   6h31m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           6h32m   True    UpToDate   6h32m

NAME                                                                         AGE     READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager   6h32m   True    UpToDate   6h32m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling           6h32m   True    UpToDate   6h32m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager   6h32m   True    UpToDate   6h32m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage           6h32m   True    UpToDate   6h32m
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kubectl delete gcp --all
iamcustomrole.iam.cnrm.cloud.google.com "gke-firewall-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier2-dnsrecord-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier2-vpcpeering-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-dnsrecord-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-firewallrule-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-subnetwork-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier3-vpcsc-admin" deleted
iamcustomrole.iam.cnrm.cloud.google.com "tier4-secretmanager-admin" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "gatekeeper-admin-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "hierarchy-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "logging-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "networking-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "policies-sa-workload-identity-binding" deleted
iampartialpolicy.iam.cnrm.cloud.google.com "projects-sa-workload-identity-binding" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-management-project-editor-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-management-project-serviceaccountadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "config-control-sa-orgroleadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "gatekeeper-admin-sa-metric-writer-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "hierarchy-sa-folderadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "logging-sa-bigqueryadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "logging-sa-logadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-dns-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-networkadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-security-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-service-control-org-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-servicedirectoryeditor-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "networking-sa-xpnadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "policies-sa-orgpolicyadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-billinguser-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectcreator-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectdeleter-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectiamadmin-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-projectmover-permissions" deleted
iampolicymember.iam.cnrm.cloud.google.com "projects-sa-serviceusageadmin-permissions" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "gatekeeper-admin-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "hierarchy-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "logging-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "networking-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "policies-sa" deleted
iamserviceaccount.iam.cnrm.cloud.google.com "projects-sa" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-629-accesscontextmanager" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-629-cloudbilling" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-629-cloudresourcemanager" deleted
service.serviceusage.cnrm.cloud.google.com "kcc-oi-629-serviceusage" deleted
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ kpt live destroy core-landing-zone
delete phase started
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket delete successful
delete phase finished
reconcile phase started
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile pending

the reconcile usually hangs

587

obriensystems commented 8 months ago

recreate

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l true -r false -d false -j false -p kcc-oi-9428
existing project: kcc-oi-9428
Date: Sun 22 Oct 2023 02:48:36 AM UTC
Timestamp: 1697942916
running with: -b kcc-oi -u oi -c false -l true -r false -d false -p kcc-oi-9428
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1697942917
unique string: oi
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc-oi4
Reusing project: kcc-oi-9428
CC_PROJECT_ID: kcc-oi-9428
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459065442144
Switching to KCC project kcc-oi-9428
Updated property [core/project].
wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER: 1020667298737
Context "gke_kcc-oi-9428_northamerica-northeast1_krmapihost-kcc-oi4" modified.
Active namespace is "config-control".
deploying core-landing-zone
get kpt release package solutions/core-landing-zone version 0.3.2
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.3.2
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.3.2 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
removing org/org-policies folder
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.3s
  Results:
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] metadata.name: set field value to "security-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.projectRef.name: set field value to "logging-project2-oi"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.name: set field value to "platform-and-component-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.projectRef.name: set field value to "logging-project2-oi"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "logging-project2-oi-security-sink"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.name: set field value to "logging-project2-oi"
    [info] spec.name: set field value to "logging-project2-oi"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] metadata.name: set field value to "dns-project2-oi-standard-core-public-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project2-oi"
    [info] spec.dnsName: set field value to "obrien.industries."
    [info] metadata.name: set field value to "dns-project2-oi"
    [info] spec.name: set field value to "dns-project2-oi"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] metadata.name: set field value to "dns-project2-oi-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] spec.projectRef.external: set field value to "kcc-oi-9428"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi"
    [info] spec.projectRef.external: set field value to "kcc-oi-9428"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.projectRef.external: set field value to "kcc-oi-9428"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/_Default"
    [info] metadata.name: set field value to "kcc-oi-9428-cloudbilling"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.name: set field value to "kcc-oi-9428-cloudresourcemanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.name: set field value to "kcc-oi-9428-serviceusage"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.name: set field value to "kcc-oi-9428-accesscontextmanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "kcc-oi-9428"
    [info] spec.member: set field value to "serviceAccount:gatekeeper-admin-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[gatekeeper-system/gatekeeper-admin]"
    [info] spec.googleServiceAccount: set field value to "gatekeeper-admin-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:hierarchy-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]"
    [info] spec.googleServiceAccount: set field value to "hierarchy-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:logging-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]"
    [info] spec.googleServiceAccount: set field value to "logging-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:service-86427388501@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "kcc-oi-9428"
    [info] spec.member: set field value to "serviceAccount:service-86427388501@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "kcc-oi-9428"
    [info] spec.member: set field value to "serviceAccount:service-86427388501@gcp-sa-yakima.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]"
    [info] spec.googleServiceAccount: set field value to "networking-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:policies-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]"
    [info] spec.googleServiceAccount: set field value to "policies-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to "serviceAccount:projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]"
    [info] spec.googleServiceAccount: set field value to "projects-sa@kcc-oi-9428.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.name: set field value to "logging-project2-oi-security-sink"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket-oi"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/security-log-bucket-oi"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage apply successful
apply phase finished
reconcile phase started
namespace/hierarchy reconcile successful
namespace/logging reconcile successful
namespace/networking reconcile successful
namespace/policies reconcile successful
namespace/projects reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed

was missing the role again - on the wrong project gcloud projects add-iam-policy-binding "${KCC_PROJECT_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer" --project "${KCC_PROJECT_ID}" --quiet

fixing

service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager   35m   False   UpdateFailed   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling           35m   False   UpdateFailed   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager   35m   False   UpdateFailed   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage           35m   False   UpdateFailed   35m
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-9428)$ kubectl get gcp | grep UpToDate
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   35m   True    UpToDate       87s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             35m   True    UpToDate       88s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     35m   True    UpToDate       83s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager   35m   True    UpToDate   25s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling           35m   True    UpToDate   25s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager   35m   True    UpToDate   25s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage           35m   True    UpToDate   25s

also fixed via

  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet

  for
 Warning  UpdateFailed  110s (x5 over 5m55s)  iampartialpolicy-controller  Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "service account 'projects/kcc-oi-9428/serviceAccounts/gatekeeper-admin-sa@kcc-oi-9428.iam.gserviceaccount.com'" with IAM Policy: Error retrieving IAM policy for service account 'projects/kcc-oi-9428/serviceAccounts/gatekeeper-admin-sa@kcc-oi-9428.iam.gserviceaccount.com': googleapi: Error 403: Permission 'iam.serviceAccounts.getIamPolicy' denied on resource (or it may not exist).
Details:

fixed
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   49m   True    UpToDate   55s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          49m   True    UpToDate   55s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            49m   True    UpToDate   55s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         49m   True    UpToDate   59s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           49m   True    UpToDate   54s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           49m   True    UpToDate   19s

do a

kpt live apply $REL_SUB_PACKAGE

to restart the update - now folders/projects coming in

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n projects
NAME                                                                              AGE     READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   6m40s   True    UpToDate   6m31s

NAME                                                                                                                   AGE     READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   6m41s   False   DependencyNotFound   6m41s
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         6m41s   False   DependencyNotFound   6m40s
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               6m41s   False   DependencyNotFound   6m40s
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      6m40s   False   DependencyNotFound   6m39s

NAME                                                                AGE     READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project2-oi       6m41s   True    UpToDate   5m44s
project.resourcemanager.cnrm.cloud.google.com/logging-project2-oi   8m20s   True    UpToDate   6m44s
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n logging
NAME                                                                                  AGE    READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi   7m7s   True    UpToDate   7m2s
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi                 7m7s   True    UpToDate   7m2s

NAME                                                                                       AGE    READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket   7m8s   True    UpToDate   7m5s
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE     READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    9m15s   True    UpToDate   8m36s
folder.resourcemanager.cnrm.cloud.google.com/clients                   9m14s   True    UpToDate   8m35s
folder.resourcemanager.cnrm.cloud.google.com/services                  9m14s   True    UpToDate   8m36s
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   9m13s   True    UpToDate   8m35s
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n policies
NAME                                                                                                         AGE     READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project   9m36s   True    UpToDate   9m8s
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get namespaces
NAME                              STATUS   AGE
cnrm-system                       Active   99m
config-control                    Active   101m
config-management-monitoring      Active   101m
config-management-system          Active   101m
configconnector-operator-system   Active   101m
default                           Active   107m
gatekeeper-system                 Active   101m
gke-gmp-system                    Active   106m
gke-managed-filestorecsi          Active   106m
gmp-public                        Active   106m
hierarchy                         Active   64m
krmapihosting-monitoring          Active   101m
krmapihosting-system              Active   104m
kube-node-lease                   Active   107m
kube-public                       Active   107m
kube-system                       Active   107m
logging                           Active   64m
networking                        Active   64m
policies                          Active   64m
projects                          Active   64m
resource-group-system             Active   101m
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n networking
No resources found in networking namespace.
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$

obriensystems commented 8 months ago

Testing setters.yaml automation in 3rd env r*@landing.systems see #591

getting https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/586 at the end

michael@cloudshell:~ (kcc-oi)$ gcloud organizations list --filter="${DIRECTORY_CUSTOMER_ID}" '--format=value(DIRECTORY_CUSTOMER_ID)'
C0...kc

testing clean install via generated setters

pull repo - switch branch to gh446-hub edit vars.sh leave setters.yaml as is - it will overwrite create empty kpt dir at root

PREFIX=ls4
KCC_PROJECT_NAME=kcc-boot-ls
SUPER_ADMIN_EMAIL=root@landing.systems
CONTACT_DOMAIN=landing.systems
CLUSTER=kcc-oi4
ROOT_FOLDER_ID=355816628468

accidentally removed the kpt folder from oi - return to ls

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ ./setup.sh -b kcc-boot-ls -u ls -n true -c true -l false -r false -d false -j false

20 min start 09:58

Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-boot-ls-8704].
Waiting for [operations/cp.7256299877501800448] to finish...done.                                                                                                                                                                     
Enabling service [cloudapis.googleapis.com] on project [kcc-boot-ls-8704]...
Operation "operations/acat.p2-145363557028-89e15456-e9b2-4179-8c46-70bb796fd14c" finished successfully.
Updated property [core/project] to [kcc-boot-ls-8704].
Updated property [core/project].
billingAccountName: billingAccounts/01E6E8-A42E99-D21FF3
billingEnabled: true
name: projects/kcc-boot-ls-8704/billingInfo
projectId: kcc-boot-ls-8704
sleep 45 sec before enabling services

Enabling APIs
Operation "operations/acf.p2-145363557028-722b7f71-9985-4d75-8642-61cab489cdb6" finished successfully.
Operation "operations/acat.p2-145363557028-d82fc815-96c8-40e7-90b3-6942f668fc97" finished successfully.
Operation "operations/acat.p2-145363557028-6075ee31-cca4-4d64-bdcd-f21841135c83" finished successfully.
Operation "operations/acat.p2-145363557028-3efda02e-0985-4592-b28b-7c31b397719c" finished successfully.
Operation "operations/acat.p2-145363557028-9bb94f02-67b7-4bb1-8dbc-8e36dc0a75bb" finished successfully.
name: organizations/59485982875/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-boot-ls-8704/global/networks/kcc-ls-vpc].
NAME: kcc-ls-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp:22,tcp:3389,icmp

Create subnet kcc-ls-sn off VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-boot-ls-8704/regions/northamerica-northeast1/subnetworks/kcc-ls-sn].
NAME: kcc-ls-sn
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
create default firewalls
Creating Anthos KCC autopilot cluster kcc-oi4 in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc on project 
Create request issued for: [kcc-oi4]
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1697983286035-6084e8a994292-fba61ecf-a1fa2177] to complete...working       

10:10
10:20
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1697983286035-6084e8a994292-fba61ecf-a1fa2177] to complete...working..                                                        
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1697983286035-6084e8a994292-fba61ecf-a1fa2177] to complete...working..                                                        
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1697983286035-6084e8a994292-fba61ecf-a1fa2177] to complete...working...                                                       
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1697983286035-6084e8a994292-fba61ecf-a1fa2177] to complete...done.                                                            
Created instance [kcc-oi4].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi4.
Cluster create time: 1105 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc-oi4.
List Clusters:
NAME: kcc-oi4
LOCATION: northamerica-northeast1
STATE: RUNNING

10:23
run lz deploy separately
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ ./setup.sh -b kcc-boot-ls -u ls -n false -c false -l true -r false -d false -j false -p kcc-boot-ls-8704

DIRECTORY_CUSTOMER_ID: C03lz5ebg
generated derived setters-core-landing-zone.yaml
./setup.sh: line 309: cd: kpt: No such file or directory

forgot to create pdt folder
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ mkdir ../../../kpt

rerun
      [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket-ls4"
    [info] spec.organizationRef.external: set field value to "59485982875"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-ls4/locations/northamerica-northeast1/buckets/security-log-bucket-ls4"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful

iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-accesscontextmanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudbilling apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudresourcemanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-serviceusage apply successful
apply phase finished
reconcile phase started
namespace/hierarchy reconcile successful
namespace/logging reconcile successful
namespace/networking reconcile successful
namespace/policies reconcile successful
namespace/projects reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-accesscontextmanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudbilling reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudresourcemanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-serviceusage reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-accesscontextmanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudresourcemanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudbilling reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-serviceusage reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile successful

1035

iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile successful
reconcile phase finished
apply phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
folder.resourcemanager.cnrm.cloud.google.com/audits apply successful
folder.resourcemanager.cnrm.cloud.google.com/clients apply successful
folder.resourcemanager.cnrm.cloud.google.com/services apply successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure apply successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-ls4 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project apply successful
apply phase finished
reconcile phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
project.resourcemanager.cnrm.cloud.google.com/logging-project-ls4 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile pending
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful

resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile successful

1041 apply phase

project.resourcemanager.cnrm.cloud.google.com/logging-project-ls4 reconcile successful
reconcile phase finished
apply phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions apply successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-ls4 apply successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-ls4 apply successful
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket apply successful
project.resourcemanager.cnrm.cloud.google.com/dns-project-ls4 apply successful
apply phase finished
reconcile phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-ls4 reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-ls4 reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile pending
project.resourcemanager.cnrm.cloud.google.com/dns-project-ls4 reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-ls4 reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-ls4 reconcile successful
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile successful

1044

project.resourcemanager.cnrm.cloud.google.com/dns-project-ls4 reconcile successful

1050
SA set
  management-project-number: "145363557028"

duration 45 min for clz

1104

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubens config-control
Context "gke_kcc-boot-ls-8704_northamerica-northeast1_krmapihost-kcc-oi4" modified.
Active namespace is "config-control".
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   31m   True    UpToDate   31m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           31m   True    UpToDate   30m

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   31m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   31m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    31m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              31m   True    UpToDate   29m

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           31m   True    UpToDate   30m

NAME                                                                               AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-accesscontextmanager   31m   True    UpToDate   31m
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudbilling           31m   True    UpToDate   31m
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudresourcemanager   31m   True    UpToDate   31m
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-serviceusage           31m   True    UpToDate   31m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get projects
No resources found in config-control namespace.
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   24m   True    UpToDate   24m

NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   24m   False   DependencyNotFound   24m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         24m   False   DependencyNotFound   24m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               24m   False   DependencyNotFound   24m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      24m   False   DependencyNotFound   24m

NAME                                                                AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-ls4       24m   True    UpToDate   21m
project.resourcemanager.cnrm.cloud.google.com/logging-project-ls4   30m   True    UpToDate   24m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    31m   True    UpToDate   28m
folder.resourcemanager.cnrm.cloud.google.com/clients                   31m   True    UpToDate   28m
folder.resourcemanager.cnrm.cloud.google.com/services                  31m   True    UpToDate   28m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   31m   True    UpToDate   28m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n logging
NAME                                                                                   AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-ls4   25m   True    UpToDate   25m
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-ls4                 25m   True    UpToDate   25m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket   25m   True    UpToDate   25m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n networking
No resources found in networking namespace.
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get namespaces
NAME                              STATUS   AGE
cnrm-system                       Active   53m
config-control                    Active   53m
config-management-monitoring      Active   54m
config-management-system          Active   54m
configconnector-operator-system   Active   54m
default                           Active   60m
gatekeeper-system                 Active   52m
gke-gmp-system                    Active   59m
gke-managed-filestorecsi          Active   59m
gmp-public                        Active   59m
hierarchy                         Active   33m
krmapihosting-monitoring          Active   54m
krmapihosting-system              Active   57m
kube-node-lease                   Active   60m
kube-public                       Active   60m
kube-system                       Active   60m
logging                           Active   33m
networking                        Active   33m
policies                          Active   33m
projects                          Active   33m
resource-group-system             Active   52m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n policies
NAME                                                                                                         AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project   32m   True    UpToDate   30m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ 

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions -n projects
Name:         mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
Namespace:    projects
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              config.k8s.io/owning-inventory: aa4fc298b6221cdddd79610cf49717502ca36ce7-1697985197779920990
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-ls4
              internal.kpt.dev/upstream-identifier:
                iam.cnrm.cloud.google.com|IAMPartialPolicy|projects|mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMPartialPolicy
Metadata:
  Creation Timestamp:  2023-10-22T14:40:40Z
  Generation:          1
  Resource Version:    33727
  UID:                 fbc7777f-bea5-4cfa-a2a5-fa5ee016be01
Spec:
  Bindings:
    Members:
      Member From:
        Log Sink Ref:
          Name:       mgmt-project-cluster-platform-and-component-log-sink
          Namespace:  logging
    Role:             roles/logging.bucketWriter
  Resource Ref:
    API Version:  resourcemanager.cnrm.cloud.google.com/v1beta1
    Kind:         Project
    Name:         logging-project-ls4
    Namespace:    projects
Status:
  Conditions:
    Last Transition Time:  2023-10-22T14:40:40Z
    Message:               reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
    Reason:                DependencyNotFound
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  DependencyNotFound  2m42s (x4 over 28m)  iampartialpolicy-controller  reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$

run 2 obrien.industries

10:10

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n true -c true -l true -r false -d false -j false 

Creating KCC project: kcc-oi-3552
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-oi-3552].
Waiting for [operations/cp.8005657574778676549] to finish...done.                                                                                                                                              
Enabling service [cloudapis.googleapis.com] on project [kcc-oi-3552]...
Operation "operations/acat.p2-850340197245-d0ef5bb4-65b7-4aaa-8449-28acb5b4f1bd" finished successfully.
Updated property [core/project] to [kcc-oi-3552].
Updated property [core/project].
billingAccountName: billingAccounts/014479-806359-2F5F85
billingEnabled: true
name: projects/kcc-oi-3552/billingInfo
projectId: kcc-oi-3552
sleep 45 sec before enabling services

10:13

Enabling APIs
Operation "operations/acf.p2-850340197245-ecdd9d29-a0a5-4569-a142-a9caf73cc3fa" finished successfully.
Operation "operations/acat.p2-850340197245-51af68f8-1012-4483-90c5-aad57dd1d024" finished successfully.
Operation "operations/acat.p2-850340197245-bf5ff1b5-7d36-4ed9-96ef-fa47e999a9e6" finished successfully.
Operation "operations/acat.p2-850340197245-81c9e54b-72da-4a1f-8c21-70aa94af0679" finished successfully.
Operation "operations/acat.p2-850340197245-9df1088d-ae65-402a-99bb-000806925895" finished successfully.
name: organizations/459065442144/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-3552/global/networks/kcc-ls-vpc].
NAME: kcc-ls-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp:22,tcp:3389,icmp

Create subnet kcc-ls-sn off VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-3552/regions/northamerica-northeast1/subnetworks/kcc-ls-sn].
NAME: kcc-ls-sn
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
create default firewalls
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc on project 
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-3552/locations/northamerica-northeast1/operations/operation-1697983973603-6084eb394b9a4-e14429ea-4173713b] to complete...working...      

10:29
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l true -r false -d true -j true -p kcc-oi-3552

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l true -r false -d true -j true -p kcc-oi-3552
existing project: kcc-oi-3552
Date: Sun 22 Oct 2023 02:29:28 PM UTC
Timestamp: 1697984968
running with: -b kcc-oi -u oi -c false -l true -r false -d true -p kcc-oi-3552
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1697984969
unique string: oi
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Reusing project: kcc-oi-3552
CC_PROJECT_ID: kcc-oi-3552
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459065442144
Switching to KCC project kcc-oi-3552
Updated property [core/project].
wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER: 850340197245
DIRECTORY_CUSTOMER_ID: C03kdhrkc
generated derived setters-core-landing-zone.yaml
deploying core-landing-zone
get kpt release package solutions/core-landing-zone version 0.3.2
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.3.2
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.3.2 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
copy over generated setting.yaml
removing org/org-policies folder
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "core-landing-zone": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
  Results:
    [info] spec.folderRef.external: set field value to "27941298022"
    [info] metadata.name: set field value to "security-log-bucket-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.projectRef.name: set field value to "logging-project-oi5"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.name: set field value to "platform-and-component-log-bucket-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.projectRef.name: set field value to "logging-project-oi5"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.resourceRef.name: set field value to "logging-project-oi5"
    [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "logging-project-oi5-security-sink"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.resourceRef.name: set field value to "logging-project-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.resourceRef.name: set field value to "logging-project-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.resourceRef.name: set field value to "logging-project-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project-oi5"
    [info] spec.resourceRef.name: set field value to "logging-project-oi5"
    [info] metadata.name: set field value to "logging-project-oi5"
    [info] spec.name: set field value to "logging-project-oi5"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] spec.folderRef.external: set field value to "27941298022"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi5"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project-oi5/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi5"
    [info] spec.folderRef.external: set field value to "27941298022"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.goog

1039 taking over triage
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-3552)$ cd ../../../kpt

  kpt alpha live plan core-landing-zone
 looks ok

      + serviceusage.cnrm.cloud.google.com/Service projects/dns-project-oi5-dns
                apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
                kind: Service
                metadata:
                  annotations:
                    cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
                    cnrm.cloud.google.com/disable-on-destroy: "false"
                    cnrm.cloud.google.com/project-id: dns-project-oi5
                    config.k8s.io/owning-inventory: cdc078bf9f321d14488babdac516a49f9574de77-1697984976342957943
                    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project-oi5
                    config.kubernetes.io/path: lz-folder/services-infrastructure/dns-project/services.yaml
                    internal.config.kubernetes.io/path: lz-folder/services-infrastructure/dns-project/services.yaml
                    internal.kpt.dev/upstream-identifier: serviceusage.cnrm.cloud.google.com|Service|projects|dns-project-id-dns
                  name: dns-project-oi5-dns
                  namespace: projects
                spec:
                  resourceID: dns.googleapis.com

expected failures due to previous lz
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp
NAME                                                                AGE   READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           16m   False   UpdateFailed   16m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   16m   False   UpdateFailed   16m

NAME                                                                                       AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   16m   False   DependencyNotReady   16m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          16m   False   DependencyNotReady   16m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            16m   False   DependencyNotReady   16m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         16m   False   DependencyNotReady   16m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           16m   False   DependencyNotReady   16m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           16m   False   DependencyNotReady   16m

NAME                                                                                                             AGE   READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   16m   False   UpdateFailed   16m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              16m   False   UpdateFailed   16m

NAME                                                              AGE   READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   16m   False   UpdateFailed   16m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          16m   False   UpdateFailed   16m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            16m   False   UpdateFailed   16m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         16m   False   UpdateFailed   16m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           16m   False   UpdateFailed   16m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           16m   False   UpdateFailed   16m

NAME                                                                          AGE   READY   STATUS         STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-accesscontextmanager   16m   False   UpdateFailed   16m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudbilling           16m   False   UpdateFailed   16m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudresourcemanager   16m   False   UpdateFailed   16m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-serviceusage           16m   False   UpdateFailed   16m

was again org level 
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa

 required permission to use project kcc-oi-3552. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=kcc-oi-3552 and then retry. Propagation of the new permission may take a few minutes.

checking sa
  management-project-number: "850340197245"

  not set

service-850340197245@gcp-sa-yakima.iam.gserviceaccount.com | Yakima Service Account for Project 850340197245 |Organization Administrator

and
gcloud projects add-iam-policy-binding "${KCC_PROJECT_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer" --project "${KCC_PROJECT_ID}" --quiet

 reconcile will continue

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp | grep False | wc -l
24
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp
NAME                                                                AGE   READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          31m   False   UpdateFailed   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       31m   True    UpToDate       98s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      31m   False   UpdateFailed   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       31m   True    UpToDate       98s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    31m   True    UpToDate       98s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      31m   True    UpToDate       98s
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           31m   True    UpToDate       98s
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   31m   True    UpToDate       98s

NAME                                                                                       AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   31m   False   DependencyNotReady   31m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          31m   True    UpToDate             83s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            31m   False   DependencyNotReady   31m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         31m   False   DependencyNotReady   31m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           31m   False   DependencyNotReady   31m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           31m   False   DependencyNotReady   31m

NAME                                                                                                             AGE   READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                31m   True    UpToDate       81s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             31m   True    UpToDate       92s
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          31m   True    UpToDate       81s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          31m   True    UpToDate       84s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       31m   True    UpToDate       69s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    31m   True    UpToDate       79s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 31m   True    UpToDate       69s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 31m   True    UpToDate       69s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   31m   False   UpdateFailed   31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              31m   False   UpdateFailed   31m

NAME                                                              AGE   READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   31m   False   UpdateFailed   31m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          31m   True    UpToDate       87s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            31m   False   UpdateFailed   31m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         31m   False   UpdateFailed   31m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           31m   False   UpdateFailed   31m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           31m   False   UpdateFailed   31m

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-accesscontextmanager   31m   True    UpToDate   2m26s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudbilling           31m   True    UpToDate   2m26s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudresourcemanager   31m   True    UpToDate   2m26s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-serviceusage           31m   True    UpToDate   2m26s
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp | grep False | wc -l
11

1102
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp | grep False | wc -l
4

kpt live to kick in the reconciler

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions
Name:         policies-sa-orgpolicyadmin-permissions
Namespace:    config-control
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              cnrm.cloud.google.com/ignore-clusterless: true
              cnrm.cloud.google.com/project-id: kcc-oi-3552
              config.k8s.io/owning-inventory: cdc078bf9f321d14488babdac516a49f9574de77-1697984976342957943
              internal.kpt.dev/upstream-identifier: iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|policies-sa-orgpolicyadmin-permissions
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMPolicyMember
Metadata:
  Creation Timestamp:  2023-10-22T14:29:45Z
  Finalizers:
    cnrm.cloud.google.com/finalizer
    cnrm.cloud.google.com/deletion-defender
  Generation:        1
  Resource Version:  44230
  UID:               8e491ecb-ff0e-4810-9a2d-d2ebec85a836
Spec:
  Member:  serviceAccount:policies-sa@kcc-oi-3552.iam.gserviceaccount.com
  Resource Ref:
    API Version:  resourcemanager.cnrm.cloud.google.com/v1beta1
    External:     459065442144
    Kind:         Organization
  Role:           roles/orgpolicy.policyAdmin
Status:
  Conditions:
    Last Transition Time:  2023-10-22T15:04:15Z
    Message:               The resource is up to date
    Reason:                UpToDate
    Status:                True
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason        Age                 From                        Message
  ----     ------        ----                ----                        -------
  Warning  UpdateFailed  26m (x20 over 55m)  iampolicymember-controller  Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "organization \"459065442144\"" with IAM Member: Role "roles/orgpolicy.policyAdmin" Member "serviceAccount:policies-sa@kcc-oi-3552.iam.gserviceaccount.com": Error retrieving IAM policy for organization "459065442144": googleapi: Error 403: Caller does not have required permission to use project kcc-oi-3552. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=kcc-oi-3552 and then retry. Propagation of the new permission may take a few minutes.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Google developer console IAM admin",
        "url": "https://console.developers.google.com/iam-admin/iam/project?project=kcc-oi-3552"
      }
    ]
  },
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/kcc-oi-3552",
      "service": "cloudresourcemanager.googleapis.com"
    },
    "reason": "USER_PROJECT_DENIED"
  }
]
, forbidden
  Warning  UpdateFailed  22m (x2 over 24m)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Error applying IAM policy for organization "459065442144": Error setting IAM policy for organization "459065442144": googleapi: Error 400: Service account policies-sa@kcc-oi-3552.iam.gserviceaccount.com does not exist., badRequest
  Normal   UpToDate      20m                iampolicymember-controller  The resource is up to date

                            michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply core-landing-zone/
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-accesscontextmanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudbilling apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudresourcemanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-serviceusage apply successful
apply phase finished
reconcile phase started
namespace/hierarchy reconcile successful
namespace/logging reconcile successful
namespace/networking reconcile successful
namespace/policies reconcile successful
namespace/projects reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-accesscontextmanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudbilling reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudresourcemanager reconcile successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-serviceusage reconcile successful
reconcile phase finished
apply phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies apply successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking apply successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
folder.resourcemanager.cnrm.cloud.google.com/audits apply successful
folder.resourcemanager.cnrm.cloud.google.com/clients apply successful
folder.resourcemanager.cnrm.cloud.google.com/services apply successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure apply successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi5 apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project apply successful
apply phase finished
reconcile phase started
rolebinding.rbac.authorization.k8s.io/allow-folders-resource-reference-to-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-config-control reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-policies reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-hierarchy-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-logging-resource-reference-from-projects reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-logging reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-networking reconcile successful
rolebinding.rbac.authorization.k8s.io/allow-projects-resource-reference-from-policies reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile pending
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi5 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile pending
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/clients reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/services reconcile successful
folder.resourcemanager.cnrm.cloud.google.com/audits reconcile successful
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi5 reconcile successful
reconcile phase finished
apply phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions apply successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi5 apply successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi5 apply successful
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket apply successful
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi5 apply successful
apply phase finished
reconcile phase started
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi5 reconcile pending
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi5 reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile pending
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi5 reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi5 reconcile successful
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-oi5 reconcile successful
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config reconcile successful
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi5 reconcile successful

missing SA's and partial policies

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#what-the-target-landing-zone-looks-like-in-gcp

obriensystems commented 8 months ago

Working hub-env automation added ambiguity on the project-parent-folder https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/596

cat << EOF > ./${REL_SUB_PACKAGE}/setters-${REL_SUB_PACKAGE}.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  org-id: "${ORG_ID}"
  project-billing-id: "${BILLING_ID}"
  project-parent-folder: project-parent-folder
  hub-project-id: ${HUB_PROJECT_ID_PREFIX}-${PREFIX}
  hub-admin: ${HUB_ADMIN_GROUP_EMAIL}
  project-allowed-restrict-vpc-peering: |
    - under:organizations/${ORG_ID}
  project-allowed-vm-external-ip-access: |
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-a/instances/fgt-primary-instance"
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-b/instances/fgt-secondary-instance"
  project-allowed-vm-can-ip-forward: |
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-a/instances/fgt-primary-instance"
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-b/instances/fgt-secondary-instance"
  fgt-primary-image: ${FORTIGATE_PRIMARY_IMAGE}
  fgt-primary-license: |
    LICENSE
  fgt-secondary-image: ${FORTIGATE_SECONDARY_IMAGE}
  fgt-secondary-license: |
    LICENSE
EOF

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l false -h true -r false -d false -j false -p kcc-oi-3552
existing project: kcc-oi-3552
Date: Mon 23 Oct 2023 03:58:16 PM UTC
Timestamp: 1698076696
running with: -b kcc-oi -u oi -c false -l false -h true -r false -d false -p kcc-oi-3552
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1698076697
unique string: oi
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Reusing project: kcc-oi-3552
CC_PROJECT_ID: kcc-oi-3552
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459065442144
Switching to KCC project kcc-oi-3552
Updated property [core/project].
wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER: 850340197245
DIRECTORY_CUSTOMER_ID: C03kdhrkc
using hub project id: xxdmu-admin1-hub-oi5
generated derived setters-hub-env.yaml
Directory kpt exists - using it

deploying hub-env
get kpt release package solutions/project/hub-env version 0.2.1
Package "hub-env":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.2.1

Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@0.2.1
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/project/hub-env/0.2.1 -> FETCH_HEAD
Adding package "solutions/project/hub-env".

Fetched 1 package(s).
copy over generated setters.yaml
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.4s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.bootDisk.initializeParams.sourceImageRef.external: set field value to "projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license"
    [info] spec.metadata[1].value: set field value to "LICENSE\n"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.bootDisk.initializeParams.sourceImageRef.external: set field value to "projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license"
    [info] spec.metadata[1].value: set field value to "LICENSE\n"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "user:michael@obrien.industries"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "serviceAccount:fortigatesdn-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"
    [info] spec.role: set field value to "organizations/459065442144/roles/FortigateSdnViewer"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.listPolicy.allow.values: set field value to "- under:organizations/459065442144\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-a/instances/fgt-primary-instance\"\n- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-b/instances/fgt-secondary-instance\"\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-a/instances/fgt-primary-instance\"\n- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-b/instances/fgt-secondary-instance\"\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "user:michael@obrien.industries"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "user:michael@obrien.industries"
    [info] metadata.name: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.name: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] spec.folderRef.name: set field value to "project-parent-folder"
    [info] metadata.name: set field value to "xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.name: set field value to "xxdmu-admin1-hub-oi5-dns"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 1.1s
  Results:
    [info] spec.metadata[2].value: Mutated field value to "|\n  config system global\n      set hostname \"fgt-ap-primary\"\n      set pre-login-banner enable\n      set admintimeout 60\n      set timezone 12\n  end\n  config system admin\n      # AC-2(A) - The Fortigates/FortiOS comes with a default local `admin` account.\n      edit \"admin\"\n          # DO NOT modify this value, it will be updated with the value in the search-replace-config.yaml\n          set password fgt-admin-password\n      next\n  end\n  config system replacemsg admin \"pre_admin-disclaimer-text\"\n      set buffer \"Acceptable Use Policy\n  WARNING: This is a private computer system. Unauthorized access or use is prohibited and subject to prosecution and/or disciplinary action. All use of this system constitutes consent to monitoring at all times and users are not entitled to any expectation of privacy. If monitoring reveals possible evidence of violation of criminal statutes, this evidence and any other related information, including identification information about the user, may be provided to law enforcement officials. If monitoring reveals violations of security regulations or unauthorized use, employees who violate security regulations or make unauthorized use of this system are subject to appropriate disciplinary action.\"\n  end\n  config router static\n      edit 10\n          set device \"port1\"\n          set gateway 172.31.200.1\n      next\n      edit 11\n        set dst 172.31.200.0/24\n        set device \"port1\"\n        set gateway 172.31.200.1\n      next\n      edit 12\n          set dst 35.191.0.0 255.255.0.0\n          set comment \"health check\"\n          set gateway 172.31.200.1\n          set device \"port1\"\n      next\n      edit 13\n          set dst 130.211.0.0 255.255.252.0\n          set comment \"health check\"\n          set gateway 172.31.200.1\n          set device \"port1\"\n      next\n      edit 20\n        set dst 172.31.201.1/32\n        set device \"port2\"\n      next\n      edit 21\n        set dst 172.31.201.0/24\n        set device \"port2\"\n        set gateway 172.31.201.1\n      next\n      edit 22\n          set dst 35.191.0.0 255.255.0.0\n          set comment \"health check\"\n          set gateway 172.31.201.1\n          set device \"port2\"\n      next\n      edit 23\n          set dst 130.211.0.0 255.255.252.0\n          set comment \"health check\"\n          set gateway 172.31.201.1\n          set device \"port2\"\n      next\n      edit 24\n          set dst 10.0.0.0 255.0.0.0\n          set comment \"route to all spokes\"\n          set gateway 172.31.201.1\n          set device \"port2\"\n      next\n      edit 30\n          set dst 172.31.203.1/32\n          set device \"port3\"\n      next\n      edit 31\n        set dst 172.31.203.0/24\n        set device \"port3\"\n        set gateway 172.31.203.1\n      next\n  end\n  config system probe-response\n      set mode http-probe\n      set http-probe-value OK\n  end\n  config system interface\n      # AC-17(100) - The allowaccess setting which enables access to the fortigate is configured to only allow SSH and HTTPS on port4 (mgmt)\n      edit port1\n          set description \"external\"\n          unset allowaccess\n          set mode static\n          set ip 172.31.200.10/32\n      next\n      edit port2\n          set description \"internal\"\n          unset allowaccess\n          set mode static\n          set ip 172.31.201.10/32\n          set explicit-web-proxy enable\n          set secondary-IP enable\n          config secondaryip\n              edit 1\n                  set ip 172.31.201.35 255.255.255.255\n              next\n          end\n      next\n      edit \"port3\"\n          set description \"transit\"\n          unset allowaccess\n          set mode static\n          set ip 172.31.203.10/32\n      next\n      edit \"port4\"\n          set description \"management\"\n          # AC-17(3) - HTTPS and SSH management access is only enabled on the mgmt interface\n          set allowaccess ping https ssh fgfm\n          set mode static\n          set ip 172.31.202.10/32\n      next\n      edit \"probe\"\n          set vdom \"root\"\n          set description \"health check probe\"\n          set allowaccess probe-response\n          set ip 169.254.255.100 255.255.255.255\n          set type loopback\n      next\n  end\n  config system ha\n      set group-name \"fgt-ap-group\"\n      set mode a-p\n      set hbdev \"port4\" 50\n      # session-pickup has impact on cpu and may be disabled to improve performance\n      set session-pickup enable\n      set ha-mgmt-status enable\n      config ha-mgmt-interfaces\n          edit 1\n              set interface \"port4\"\n              set gateway 172.31.202.1\n          next\n      end\n      set override enable\n      set priority 200\n      set unicast-hb enable\n      set unicast-hb-peerip 172.31.202.11\n      set unicast-hb-netmask 255.255.255.0\n  end\n  config system sdn-connector\n      edit \"gcp\"\n          set type gcp\n          set ha-status enable\n      next\n  end\n  config system dns\n    set primary 169.254.169.254\n    set protocol cleartext\n    unset secondary\n  end\n  # Everything underneath this line will be synchronised to the secondary node with HA\n  # Explicit proxy for APPRZ and DATARZ workloads\n  config system settings\n      set gui-explicit-proxy enable\n  end\n  config web-proxy explicit\n      set status enable\n      set http-incoming-port 8080\n      set https-incoming-port 8080\n  end\n  # Internal Load balancers health check\n  ## VIP\n  config firewall vip\n        edit \"ilb-healthcheck-vip\"\n          set extip 172.31.201.30\n          set mappedip \"169.254.255.100\"\n          set extintf \"port2\"\n          set portforward enable\n          set extport 8008\n          set mappedport 8008\n      next\n      edit \"ilb-proxy-healthcheck-vip\"\n          set extip 172.31.201.35\n          set mappedip \"169.254.255.100\"\n          set extintf \"port2\"\n          set portforward enable\n          set extport 8008\n          set mappedport 8008\n      next\n  end\n  ## VIP Group\n  config firewall vipgrp\n      edit \"ilb-healthcheck-vipgrp\"\n          set interface \"port2\"\n          set member \"ilb-healthcheck-vip\" \"ilb-proxy-healthcheck-vip\"\n          set comment \"This group contains VIP objects representing internal load balancers health checks. It is referenced in a policy forwarding traffic to the probe loopback interface\"\n      next\n  end\n  ## Service\n  config firewall service custom\n      edit \"PROBE\"\n          set tcp-portrange 8008\n      next\n  end\n  ## Policy\n  config firewall policy\n      edit 0\n          set name \"ilb healthcheck\"\n          set srcintf \"port2\"\n          set dstintf \"probe\"\n          set action accept\n          set srcaddr \"all\"\n          set dstaddr \"ilb-healthcheck-vipgrp\"\n          set schedule \"always\"\n          set service \"PROBE\"\n          set comment \"This policy forwards internal load balancers health checks to the probe loopback interface\"\n      next\n  end"
    [info] spec.metadata[2].value: Mutated field value to "|\n  config system global\n      set hostname \"fgt-ap-secondary\"\n      set pre-login-banner enable\n      set admintimeout 60\n      set timezone 12\n  end\n  config system admin\n      # AC-2(A) - The Fortigates/FortiOS comes with a default local `admin` account.\n      edit \"admin\"\n          # DO NOT modify this value, it will be updated with the value in the search-replace-config.yaml\n          set password fgt-admin-password\n      next\n  end\n  config system replacemsg admin \"pre_admin-disclaimer-text\"\n      set buffer \"Acceptable Use Policy\n  WARNING: This is a private computer system. Unauthorized access or use is prohibited and subject to prosecution and/or disciplinary action. All use of this system constitutes consent to monitoring at all times and users are not entitled to any expectation of privacy. If monitoring reveals possible evidence of violation of criminal statutes, this evidence and any other related information, including identification information about the user, may be provided to law enforcement officials. If monitoring reveals violations of security regulations or unauthorized use, employees who violate security regulations or make unauthorized use of this system are subject to appropriate disciplinary action.\"\n  end\n  config router static\n      edit 10\n          set device \"port1\"\n          set gateway 172.31.200.1\n      next\n      edit 11\n        set dst 172.31.200.0/24\n        set device \"port1\"\n        set gateway 172.31.200.1\n      next\n      edit 12\n          set dst 35.191.0.0 255.255.0.0\n          set comment \"health check\"\n          set gateway 172.31.200.1\n          set device \"port1\"\n      next\n      edit 13\n          set dst 130.211.0.0 255.255.252.0\n          set comment \"health check\"\n          set gateway 172.31.200.1\n          set device \"port1\"\n      next\n      edit 20\n        set dst 172.31.201.1/32\n        set device \"port2\"\n      next\n      edit 21\n        set dst 172.31.201.0/24\n        set device \"port2\"\n        set gateway 172.31.201.1\n      next\n      edit 22\n          set dst 35.191.0.0 255.255.0.0\n          set comment \"health check\"\n          set gateway 172.31.201.1\n          set device \"port2\"\n      next\n      edit 23\n          set dst 130.211.0.0 255.255.252.0\n          set comment \"health check\"\n          set gateway 172.31.201.1\n          set device \"port2\"\n      next\n      edit 24\n          set dst 10.0.0.0 255.0.0.0\n          set comment \"route to all spokes\"\n          set gateway 172.31.201.1\n          set device \"port2\"\n      next\n      edit 30\n          set dst 172.31.203.1/32\n          set device \"port3\"\n      next\n      edit 31\n        set dst 172.31.203.0/24\n        set device \"port3\"\n        set gateway 172.31.203.1\n      next\n  end\n  config system probe-response\n      set mode http-probe\n      set http-probe-value OK\n  end\n  config system interface\n      # AC-17(100) - The allowaccess setting which enables access to the fortigate is configured to only allow SSH and HTTPS on port4 (mgmt)\n      edit port1\n          set description \"external\"\n          unset allowaccess\n          set mode static\n          set ip 172.31.200.11/32\n      next\n      edit port2\n          set description \"internal\"\n          unset allowaccess\n          set mode static\n          set ip 172.31.201.11/32\n          set explicit-web-proxy enable\n          set secondary-IP enable\n          config secondaryip\n              edit 1\n                  set ip 172.31.201.35 255.255.255.255\n              next\n          end\n      next\n      edit \"port3\"\n          set description \"transit\"\n          unset allowaccess\n          set mode static\n          set ip 172.31.203.11/32\n      next\n      edit \"port4\"\n          set description \"management\"\n          # AC-17(3) - HTTPS and SSH management access is only enabled on the mgmt interface\n          set allowaccess ping https ssh fgfm\n          set mode static\n          set ip 172.31.202.11/32\n      next\n      edit \"probe\"\n          set vdom \"root\"\n          set description \"health check probe\"\n          set allowaccess probe-response\n          set ip 169.254.255.100 255.255.255.255\n          set type loopback\n      next\n  end\n  config system ha\n      set group-name \"fgt-ap-group\"\n      set mode a-p\n      set hbdev \"port4\" 50\n      # session-pickup has impact on cpu and may be disabled to improve performance\n      set session-pickup enable\n      set ha-mgmt-status enable\n      config ha-mgmt-interfaces\n          edit 1\n              set interface \"port4\"\n              set gateway 172.31.202.1\n          next\n      end\n      set override enable\n      set priority 100\n      set unicast-hb enable\n      set unicast-hb-peerip 172.31.202.10\n      set unicast-hb-netmask 255.255.255.0\n  end\n  config system sdn-connector\n      edit \"gcp\"\n          set type gcp\n          set ha-status enable\n      next\n  end\n  config system dns\n    set primary 169.254.169.254\n    set protocol cleartext\n    unset secondary\n  end\n  # explicit proxy for APPRZ and DATARZ workloads\n  config system settings\n      set gui-explicit-proxy enable\n  end\n  config web-proxy explicit\n      set status enable\n      set http-incoming-port 8080\n      set https-incoming-port 8080\n  end"

Successfully executed 2 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
error: invalid object: "projects_xxdmu-admin1-hub-oi5_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5 -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

obriensystems commented 8 months ago

triaging

error: invalid object: "projects_xxdmu-admin1-hub-oi5_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5 -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

missing parent folder
data:
  org-id: "459065442144"
  project-billing-id: "014479-806359-2F5F85"
  project-parent-folder: project-parent-folder

  set
    project-parent-folder: #{HUB_PROJECT_PARENT_FOLDER}
    via
  HUB_PROJECT_PARENT_FOLDER=services-infrastructure

  rerun
     [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 400ms
  Results:
    [info]: no matches

Successfully executed 2 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
error: invalid object: "projects_xxdmu-admin1-hub-oi5_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5 -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

try folder id

  #  project-parent-folder: services-infrastructure
  project-parent-folder: 1029814987930

ichael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-3552)$ cd ../../../kpt/
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt fn render hub-env/ --truncate-output=false
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.bootDisk.initializeParams.sourceImageRef.external: set field value to "projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license"
    [info] spec.metadata[1].value: set field value to "LICENSE\n"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.bootDisk.initializeParams.sourceImageRef.external: set field value to "projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license"
    [info] spec.metadata[1].value: set field value to "LICENSE\n"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "user:michael@obrien.industries"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "serviceAccount:fortigatesdn-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"
    [info] spec.role: set field value to "organizations/459065442144/roles/FortigateSdnViewer"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "serviceusage.cnrm.cloud.google.com/namespaces/projects/Service/xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.listPolicy.allow.values: set field value to "- under:organizations/459065442144\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-a/instances/fgt-primary-instance\"\n- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-b/instances/fgt-secondary-instance\"\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-a/instances/fgt-primary-instance\"\n- \"projects/xxdmu-admin1-hub-oi5/zones/northamerica-northeast1-b/instances/fgt-secondary-instance\"\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "user:michael@obrien.industries"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.member: set field value to "user:michael@obrien.industries"
    [info] metadata.name: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.name: set field value to "xxdmu-admin1-hub-oi5"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] spec.folderRef.name: set field value to "1029814987930"
    [info] metadata.name: set field value to "xxdmu-admin1-hub-oi5-compute"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
    [info] metadata.name: set field value to "xxdmu-admin1-hub-oi5-dns"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi5"
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 400ms
  Results:
    [info]: no matches

Successfully executed 2 function(s) in 1 package(s).

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
error: invalid object: "projects_xxdmu-admin1-hub-oi5_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5 -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

original code

apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"

2nd mod https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/commit/25fee09dd6c62931032569fbc2cc8bf090fd9266#diff-0453fd24870bbd1f648b0d14dcbd3877e4eceafba5cf535c1b962ac63969a94f

apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"

what I have from Aug 15 working in kcc.landing.systems

apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'

we are good in hub-env/project.yaml

spec:
  name: xxdmu-admin1-hub-oi5 # kpt-set: ${hub-project-id}
  billingAccountRef:
    external: "014479-806359-2F5F85" # kpt-set: ${project-billing-id}
  folderRef:
    name: "services-infrastructure" # kpt-set: ${project-parent-folder}
    namespace: hierarchy

ot picking up the change after render/apply

found it - I forgot I fixed a hardcoded depends-on dependency in project.yaml off hub-env

  annotations:
    cnrm.cloud.google.com/auto-create-network: "false"
    #   config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure
    #config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/176411558066
    internal.kpt.dev/upstream-identifier: 'resourcemanager.cnrm.cloud.google.com|Project|projects|hub-project-id'
    cnrm.cloud.google.com/blueprint: 'kpt-pkg-fn-live'
spec:
  name: dmu-admin1-hub-kls # kpt-set: ${hub-project-id}

working

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 apply successful
apply phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending

obriensystems commented 8 months ago

seeing a hang now on "kpt live apply" https://github.com/kptdev/kpt/issues/825

however a 2nd run restarted after 5 min

ichael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 apply successful
apply phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile successful
reconcile phase finished
apply phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns apply successful
apply phase finished
reconcile phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile successful

iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile failed
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile successful
reconcile phase finished
apply phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply skipped: dependency apply reconcile failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply skipped: dependency apply reconcile failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
apply phase finished
reconcile phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile skipped
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile skipped
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile skipped

4 min in

computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
reconcile phase finished
apply phase started
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr apply successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route apply successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply skipped: dependency apply actuation skipped: networking_hub-fortigatesdn-sa_iam.cnrm.cloud.google.com_IAMServiceAccount
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply skipped: dependency apply actuation skipped: networking_hub-managementvm-sa_iam.cnrm.cloud.google.com_IAMServiceAccount
apply phase finished
reconcile phase started
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr reconcile pending
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile pending
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile skipped
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile successful

obriensystems commented 8 months ago

starting to come in but no hub project yet

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp
NAME                                                                  AGE     READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin            27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role   4m21s   True    UpToDate   4m20s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin         27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin        27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin         27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin      27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin        27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin             27h     True    UpToDate   140m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin     27h     True    UpToDate   26h

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           27h   True    UpToDate   26h

NAME                                                                                                             AGE     READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions                             4m21s   True    UpToDate       4m7s
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions                        4m21s   True    UpToDate       4m6s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         4m20s   False   UpdateFailed   4m7s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          4m20s   False   UpdateFailed   4m6s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           4m20s   False   UpdateFailed   4m5s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              27h     True    UpToDate       26h

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           27h   True    UpToDate   25h
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           27h   True    UpToDate   26h

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-accesscontextmanager   27h   True    UpToDate   26h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudbilling           27h   True    UpToDate   26h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudresourcemanager   27h   True    UpToDate   26h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-serviceusage           27h   True    UpToDate   26h

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   26h   True    UpToDate   26h

NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   26h   False   DependencyNotFound   26h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         26h   False   DependencyNotFound   26h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               26h   False   DependencyNotFound   26h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      26h   False   DependencyNotFound   26h

NAME                                                                 AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi5        26h   True    UpToDate   26h
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi5    26h   True    UpToDate   26h
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5   16m   True    UpToDate   5m10s

NAME                                                                      AGE    READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute   5m7s   True    UpToDate   5m4s
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns       5m7s   True    UpToDate   4m43s

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n policies
NAME                                                                                                                               AGE     READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project                  5m47s   True    UpToDate   5m46s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project                         5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project                         26h     True    UpToDate   26h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project   5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project                        5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project                      5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project                           5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project                       5m45s   True    UpToDate   5m44s

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    26h   True    UpToDate   26h
folder.resourcemanager.cnrm.cloud.google.com/clients                   26h   True    UpToDate   26h
folder.resourcemanager.cnrm.cloud.google.com/services                  26h   True    UpToDate   26h
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   26h   True    UpToDate   26h

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                                               AGE     READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                               6m11s   True    UpToDate             5m59s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   6m9s    False   DependencyNotFound   6m9s

NAME                                                                   AGE     READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   6m56s   True    UpToDate   6m22s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   6m56s   True    UpToDate   54s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       6m56s   True    UpToDate   3s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    6m56s   True    UpToDate   6m22s

NAME                                                                    AGE     READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   6m10s   True    UpToDate   5m57s

NAME                                                                                AGE     READY   STATUS     STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   6m10s   True    UpToDate   5m59s

NAME                                                                          AGE     READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        6m10s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    6m10s   True    UpToDate   5m57s

NAME                                                                 AGE     READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    6m10s   True    UpToDate   6m9s

obriensystems commented 8 months ago

status

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                                               AGE     READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                               6m11s   True    UpToDate             5m59s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   6m9s    False   DependencyNotFound   6m9s

NAME                                                                   AGE     READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   6m56s   True    UpToDate   6m22s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   6m56s   True    UpToDate   54s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       6m56s   True    UpToDate   3s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    6m56s   True    UpToDate   6m22s

NAME                                                                    AGE     READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   6m10s   True    UpToDate   5m57s

NAME                                                                                AGE     READY   STATUS     STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   6m10s   True    UpToDate   5m59s

NAME                                                                          AGE     READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        6m10s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    6m10s   True    UpToDate   5m57s

NAME                                                                 AGE     READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    6m10s   True    UpToDate   6m9s

firewall depends on hub-fortigatesdn-sa service account
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr -n networking
Status:
  Conditions:
    Last Transition Time:  2023-10-23T17:33:52Z
    Message:               reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found

check the service account

obriensystems commented 8 months ago

Reproducing error Chris C. found where we are missing the management-project-id: from setters.yaml Thank you Chris for this 3rd fix for the hub-env

member: "serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com" # kpt-set: serviceAccount:networking-sa@${management-project-id}.iam.gserviceaccount.com

need to look for it first so I can be sure to fix it

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp | grep False
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         174m   False   UpdateFailed   173m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          174m   False   UpdateFailed   173m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           174m   False   UpdateFailed   173m

all 3 are
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions 

  Warning  UpdateFailed  72s (x88 over 172m)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project "xxdmu-admin1-hub-oi5"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@management-project-id.iam.gserviceaccount.com for project \"xxdmu-admin1-hub-oi5\"" both failed. Final error: Error applying IAM policy for project "xxdmu-admin1-hub-oi5": Error setting IAM policy for project "xxdmu-admin1-hub-oi5": googleapi: Error 400: Service account networking-sa@management-project-id.iam.gserviceaccount.com does not exist., badRequest

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   174m   False   DependencyNotFound   174m

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl describe computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr  -n networking
Status:
  Conditions:
    Last Transition Time:  2023-10-23T17:33:52Z
    Message:               reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found

    all above except below

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl describe computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr   -n networking

  Warning  DependencyNotReady  32m                    computefirewall-controller  reference ComputeNetwork networking/hub-global-mgmt-vpc is not ready
  Warning  DependencyNotFound  4m10s (x17 over 179m)  computefirewall-controller  reference IAMServiceAccount networking/hub-managementvm-sa is not found

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp -n projects | grep False
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29h   False   DependencyNotFound   29h

rerunning after adding
  management-project-id: ${HUB_PROJECT_ID_PREFIX}-${PREFIX}

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u oi -n false -c false -l false -h true -r false -d false -j false -p kcc-oi-3552

uccessfully executed 2 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 apply successful
apply phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile successful
reconcile phase finished
apply phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi5\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"config.kubernetes.io/depends-on\":\"resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|networking-sa-computeinstanceadmin-permissions\"},\"name\":\"networking-sa-computeinstanceadmin-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-hub-oi5\",\"kind\":\"Project\"},\"role\":\"roles/compute.instanceAdmin.v1\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-computeinstanceadmin-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi5\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"config.kubernetes.io/depends-on\":\"resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|networking-sa-serviceaccountadmin-permissions\"},\"name\":\"networking-sa-serviceaccountadmin-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-hub-oi5\",\"kind\":\"Project\"},\"role\":\"roles/iam.serviceAccountAdmin\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-serviceaccountadmin-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi5\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"config.kubernetes.io/depends-on\":\"resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|networking-sa-serviceaccountuser-permissions\"},\"name\":\"networking-sa-serviceaccountuser-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-hub-oi5\",\"kind\":\"Project\"},\"role\":\"roles/iam.serviceAccountUser\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-serviceaccountuser-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns apply successful
apply phase finished
reconcile phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile successful
reconcile phase finished
apply phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply skipped: dependency apply actuation failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply skipped: dependency apply actuation failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
apply phase finished
reconcile phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile skipped
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile skipped
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile skipped
reconcile phase finished
apply phase started
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr apply successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route apply successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply skipped: dependency apply actuation skipped: networking_hub-fortigatesdn-sa_iam.cnrm.cloud.google.com_IAMServiceAccount
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply skipped: dependency apply actuation skipped: networking_hub-managementvm-sa_iam.cnrm.cloud.google.com_IAMServiceAccount
apply phase finished
reconcile phase started
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr reconcile pending
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile skipped

deleting - as dmu project says up in k8s but it is not showing

uccessfully executed 2 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 apply successful
apply phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile successful
reconcile phase finished
apply phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi5\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"config.kubernetes.io/depends-on\":\"resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|networking-sa-computeinstanceadmin-permissions\"},\"name\":\"networking-sa-computeinstanceadmin-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-hub-oi5\",\"kind\":\"Project\"},\"role\":\"roles/compute.instanceAdmin.v1\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-computeinstanceadmin-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi5\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"config.kubernetes.io/depends-on\":\"resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|networking-sa-serviceaccountadmin-permissions\"},\"name\":\"networking-sa-serviceaccountadmin-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-hub-oi5\",\"kind\":\"Project\"},\"role\":\"roles/iam.serviceAccountAdmin\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-serviceaccountadmin-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi5\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"config.kubernetes.io/depends-on\":\"resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|config-control|networking-sa-serviceaccountuser-permissions\"},\"name\":\"networking-sa-serviceaccountuser-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-hub-oi5\",\"kind\":\"Project\"},\"role\":\"roles/iam.serviceAccountUser\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@xxdmu-admin1-hub-oi5.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-serviceaccountuser-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns apply successful
apply phase finished
reconcile phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile successful
reconcile phase finished
apply phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply skipped: dependency apply actuation failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply skipped: dependency apply actuation failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply skipped: dependency apply actuation failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
apply phase finished
reconcile phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile skipped
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile skipped
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile skipped
reconcile phase finished
apply phase started
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr apply successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route apply successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply skipped: dependency apply actuation skipped: networking_hub-fortigatesdn-sa_iam.cnrm.cloud.google.com_IAMServiceAccount
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply skipped: dependency apply actuation skipped: networking_hub-managementvm-sa_iam.cnrm.cloud.google.com_IAMServiceAccount
apply phase finished
reconcile phase started
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr reconcile pending
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile skipped

computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
reconcile phase finished
delete phase started
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns delete successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project delete successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project delete successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions delete successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions delete successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions delete successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions delete successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions delete successful
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role delete successful
delete phase finished
reconcile phase started
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
reconcile phase finished
delete phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 delete successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions delete successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance delete successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance delete successful
delete phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 40 attempted, 40 successful, 0 skipped, 0 failed
reconcile result: 40 attempted, 40 successful, 0 skipped, 0 failed, 0 timed out

redeploy
metadata: # kpt-merge: config-control/fortigatesdn-sa-fortigatesdnviewer-role-permissions
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}
  annotations:
    cnrm.cloud.google.com/project-id: xxdmu-admin1-hub-oi5 # kpt-set: ${hub-project-id}
    cnrm.cloud.google.com/ignore-clusterless: "true"
    #config.kubernetes.io/depends-on: iam.cnrm.cloud.google.com/namespaces/networking/#IAMServiceAccount/hub-fortigatesdn-sa
    #internal.kpt.dev/upstream-identifier: 'iam.cnrm.cloud.google.com|IAMPolicyMember|#config-control|fortigatesdn-sa-fortigatesdnviewer-role-permissions'
    #cnrm.cloud.google.com/blueprint: 'kpt-pkg-fn-live'

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile failed

project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile failed

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n projects | grep False
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29h   False   DependencyNotFound   29h
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5   3m40s   False   UpdateFailed   3m39s

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 -n projects

  Warning  UpdateFailed  118s (x7 over 4m8s)  project-controller  Update call failed: error applying desired state: summary: error creating project xxdmu-admin1-hub-oi5 (xxdmu-admin1-hub-oi5): googleapi: Error 409: Requested entity already exists, alreadyExists. If you received a 403 error, make sure you have the `roles/resourcemanager.projectCreator` permission

ichael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live destroy hub-env

renaming project
  project-parent-folder: services-infrastructure
  hub-project-id: xxdmu-admin1-hub-oi6
  management-project-id: xxdmu-admin1-hub-oi6

render

set
    cnrm.cloud.google.com/project-id: xxdmu-admin1-hub-oi6 # kpt-set: ${hub-project-id}

apply

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile failed

<img width="1285" alt="Screenshot 2023-10-23 at 17 04 27" src="https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/564a6ab9-9d79-456d-81ee-8ef87cea7842">

project now up

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n projects | grep False
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29h   False   DependencyNotFound   29h
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6   34s   False   Updating   33s
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n projects | grep False
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               29h   False   DependencyNotFound   29h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      29h   False   DependencyNotFound   29h

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp | grep False
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions                    4m13s   False   UpdateFailed   4m12s

  Warning  UpdateFailed  25s (x8 over 4m38s)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Error applying IAM policy for organization "459065442144": Error setting IAM policy for organization "459065442144": googleapi: Error 400: Service account fortigatesdn-sa@xxdmu-admin1-hub-oi6.iam.gserviceaccount.com does not exist., badRequest

edit depends-on - additional

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed

VPC issues

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live destroy hub-env
delete phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 delete successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa delete successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa delete successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions delete successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions delete successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance delete successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance delete successful
delete phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 7 attempted, 7 successful, 0 skipped, 0 failed
reconcile result: 7 attempted, 7 successful, 0 skipped, 0 failed, 0 timed out

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi6 reconcile failed

forgot to rename the project


render
apply

  hub-project-id: xxdmu-admin1-hub-oi7
  management-project-id: xxdmu-admin1-hub-oi7

create group sas - stop using user: member: "user:michael@obrien.industries" # kpt-set: ${hub-admin}


michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-pkg-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-hub-oi7\",\"config.k8s.io/owning-inventory\":\"b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809\",\"internal.kpt.dev/upstream-identifier\":\"iam.cnrm.cloud.google.com|IAMPolicyMember|networking|hub-admin-serviceaccountuser-permissions\"},\"name\":\"hub-admin-serviceaccountuser-permissions\",\"namespace\":\"networking\"},\"spec\":{\"member\":\"group:sas@obrien.industries\",\"resourceRef\":{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMServiceAccount\",\"name\":\"hub-managementvm-sa\"},\"role\":\"roles/iam.serviceAccountUser\"}}\n"}},"spec":{"member":"group:sas@obrien.industries"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "hub-admin-serviceaccountuser-permissions", Namespace: "networking"
for: "fortigate/management-vm/service-account.yaml": error when patching "fortigate/management-vm/service-account.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi7 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi7 reconcile successful

SA requires delete

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live destroy hub-env
delete phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi7 delete successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa delete successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa delete successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions delete successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions delete successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance delete successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance delete successful
delete phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi7 reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi7 reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 7 attempted, 7 successful, 0 skipped, 0 failed
reconcile result: 7 attempted, 7 successful, 0 skipped, 0 failed, 0 timed out

too fast

ame: "hub-managementvm-sa", Namespace: "networking"
for: "fortigate/management-vm/service-account.yaml": error when patching "fortigate/management-vm/service-account.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot make changes to container annotation cnrm.cloud.google.com/project-id
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi8 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile skipped
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile skipped
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi8 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi8 reconcile successful

destroy - try 9

Successfully executed 2 function(s) in 1 package(s).
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi9 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi9 reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed

obriensystems commented 8 months ago

SAs for hub are in the networking namespace instead of the config-control one - one of 3 resources is left in config-control - fix is to remove the kpt-set and hardcode to networking

metadata: # kpt-merge: networking/hub-managementvm-sa
  name: hub-managementvm-sa
  namespace: networking

metadata: # kpt-merge: networking/hub-fortigatesdn-sa
  name: hub-fortigatesdn-sa
  namespace: networking

found the issue 
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}

should be networking namespace

this would fix the fact that the IAMPolicyMember is in the config-control namespace but it's service account is in the networking namespace

to summarize
hub-env/fortigate/service-account.yaml
- sa in networking
- policy in config-control

hub-env/fortigate/management-vm/service-account.yaml
- sa in networking
- policy in networking

fix is to move the 
fortigatesdn-sa-fortigatesdnviewer-role-permissions
to the network namespace

metadata: # kpt-merge: config-control/fortigatesdn-sa-fortigatesdnviewer-role-permissions
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}

to
metadata: # kpt-merge: networking/fortigatesdn-sa-fortigatesdnviewer-role-permissions
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}

to match
metadata: # kpt-merge: networking/hub-admin-serviceaccountuser-permissions
  name: hub-admin-serviceaccountuser-permissions
  namespace: networking

and add
 management-namespace: networking
to setters.yaml

add to setters

  management-namespace: networking

fixes
  namespace: networking # kpt-set: ${management-namespace}

but breaks the VM's - we will hardcode instead like the management-vm/service-account.yaml

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
error: 5 errors:
- invalid object: "networking_hub-fgt-primary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-primary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-fgt-secondary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-secondary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-httphc_compute.cnrm.cloud.google.com_ComputeHTTPHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHTTPHealthCheck/hub-http-8008-httphc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-hc_compute.cnrm.cloud.google.com_ComputeHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHealthCheck/hub-http-8008-hc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-mgmt-data-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-mgmt-data-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
and also hardcoded ones in project-iam.yaml
  namespace: config-control # kpt-set: ${management-namespace}

fixed

  michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
error: 5 errors:
- invalid object: "networking_hub-fgt-primary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-primary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-fgt-secondary-log-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-fgt-secondary-log-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-httphc_compute.cnrm.cloud.google.com_ComputeHTTPHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHTTPHealthCheck/hub-http-8008-httphc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-http-8008-hc_compute.cnrm.cloud.google.com_ComputeHealthCheck": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeHealthCheck/hub-http-8008-hc -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
- invalid object: "networking_hub-mgmt-data-disk_compute.cnrm.cloud.google.com_ComputeDisk": invalid "config.kubernetes.io/depends-on" annotation: external dependency: compute.cnrm.cloud.google.com/namespaces/networking/ComputeDisk/hub-mgmt-data-disk -> iam.cnrm.cloud.google.com/namespaces/config-control/IAMPolicyMember/networking-sa-computeinstanceadmin-permissions
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt fn render hub-env
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-hub-oi9"
    ...(113 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 400ms
  Results:
    [info]: no matches

Successfully executed 2 function(s) in 1 package(s).
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi9 apply successful
apply phase finished
reconcile phase started
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi9 reconcile successful

destroy
recreate

moved over

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                       AGE   READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     79s   False   DependencyNotFound   79s
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   79s   False   DependencyNotFound   79s

NAME                                                                                            AGE   READY   STATUS               STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   82s   False   UpdateFailed         81s
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              81s   False   DependencyNotReady   81s

NAME                                                              AGE   READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   81s   False   UpdateFailed   81s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   81s   False   UpdateFailed   81s

further
  Warning  DependencyNotFound  2m19s  computeinstance-controller  reference ComputeDisk networking/hub-fgt-primary-log-disk is not found
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance  -n networking

  Warning  UpdateFailed  75s (x7 over 3m22s)  iampolicymember-controller  Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "organization \"459065442144\"" with IAM Member: Role "organizations/459065442144/roles/FortigateSdnViewer" Member "serviceAccount:fortigatesdn-sa@xxdmu-admin1-hub-oi10.iam.gserviceaccount.com": Error retrieving IAM policy for organization "459065442144": googleapi: Error 403: The caller does not have permission, forbidden

need to add Organization Role Admin (and update the script to retrieve the account set the role before rendering

Status:
  Conditions:
    Last Transition Time:  2023-10-23T23:11:05Z
    Message:               Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).

also need to add "Create Service Accounts" role

destroy/recreate because of immutable SAs

apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  org-id: "459065442144"
  project-billing-id: "014479-806359-2F5F85"
  project-parent-folder: services-infrastructure
  hub-project-id: xxdmu-admin1-hub-oi11
  management-project-id: xxdmu-admin1-hub-oi11
  # must be config-control
  management-namespace: config-control
  hub-admin: group:sas@obrien.industries
  project-allowed-restrict-vpc-peering: |
    - under:organizations/459065442144
  project-allowed-vm-external-ip-access: |
    - "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
    - "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
  project-allowed-vm-can-ip-forward: |
    - "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
    - "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
  fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
  fgt-primary-license: |
    LICENSE
  fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
  fgt-secondary-license: |
    LICENSE

obriensystems commented 8 months ago

The networking-sa service account used by the hub-env requires the role roles/iam.serviceAccountAdmin Fix was to add the same permissions as for core-landing-zone in the config-control namespace to the hub-env in the networking namespace to the networking-sa service account

before
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                       AGE    READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     128m   False   DependencyNotFound   128m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   128m   False   DependencyNotFound   128m

NAME                                                                                            AGE    READY   STATUS               STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   128m   False   UpdateFailed         128m
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              128m   False   DependencyNotReady   128m

NAME                                                              AGE    READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   128m   False   UpdateFailed   128m
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   128m   False   UpdateFailed   128m

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud projects list --filter="kcc-oi-3552" '--format=value(PROJECT_NUMBER)'
850340197245

iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   119m   False   UpdateFailed   119m
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa  -n networking

    Message:               Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).

service-850340197245@gcp-sa-yakima.iam.gserviceaccount.com | Yakima Service Account for Project 850340197245 | Organization AdministratorOrganization Role AdministratorService Account Admin
-- | -- | --

delete it and wait for recreation after an apply
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl delete iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa  -n networking
iamserviceaccount.iam.cnrm.cloud.google.com "hub-fortigatesdn-sa" deleted

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
...
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile failed

iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   51s    False   UpdateFailed   51s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   131m   False   UpdateFailed   131m

another approach to get the permission set per project instead of org

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ export SA_EMAIL="$(kubectl get ConfigConnectorContext -n networking -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ echo $SA_EMAIL
networking-sa@kcc-oi-3552.iam.gserviceaccount.com

this one

networking-sa@kcc-oi-3552.iam.gserviceaccount.com | networking-sa | Access Context Manager AdminCompute Shared VPC Admin
-- | -- | --

add role
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet > /dev/null 1>&1
Updated IAM policy for organization [459065442144].

from

networking-sa@kcc-oi-3552.iam.gserviceaccount.com | networking-sa | Access Context Manager AdminCompute Shared VPC AdminService Account Admin
-- | -- | --

fixed without any delete/render
  Warning  UpdateFailed  2m26s (x8 over 6m34s)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "iam.googleapis.com",
    "metadata": {
      "permission": "iam.serviceAccounts.create"
    },
    "reason": "IAM_PERMISSION_DENIED"
  }
]
, forbidden
  Normal  Updating  26s (x9 over 6m34s)  iamserviceaccount-controller  Update in progress
  Normal  UpToDate  24s                  iamserviceaccount-controller  The resource is up to date

  fixed
  NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   7m35s   True    UpToDate   84s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   147m    True    UpToDate   51s

after
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                           AGE   READY   STATUS         STATUS AGE
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk   17m   False   UpdateFailed   17m

NAME                                                                       AGE    READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     148m   False   DependencyNotFound   148m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   148m   False   DependencyNotFound   148m

NAME                                                                                            AGE    READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   148m   False   UpdateFailed   148m

these 3 are fixed in the queue
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              148m   True    UpToDate       104s

NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   8m33s   True    UpToDate   2m22s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   148m    True    UpToDate   109s

obriensystems commented 8 months ago

spawned https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/602

moving on to remaining dependencies
ichael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking | grep False
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk   24m   False   UpdateFailed   24m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     155m   False   DependencyNotFound   155m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   155m   False   DependencyNotFound   155m
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   155m   False   UpdateFailed   155m

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions -n networking

    Message:               Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "organization \"459065442144\"" with IAM Member: Role "organizations/459065442144/roles/FortigateSdnViewer" Member "serviceAccount:fortigatesdn-sa@xxdmu-admin1-hub-oi11.iam.gserviceaccount.com": Error retrieving IAM policy for organization "459065442144": googleapi: Error 403: The caller does not have permission, forbidden

it needs the same as in core-landing-zone
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.organizationRoleAdmin --condition=None --quiet > /dev/null 1>&1

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.organizationRoleAdmin --condition=None --quiet > /dev/null 1>&1
Updated IAM policy for organization [459065442144].

before

after

update not picking up - trying delete
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl delete iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions  -n networking
iampolicymember.iam.cnrm.cloud.google.com "fortigatesdn-sa-fortigatesdnviewer-role-permissions" deleted

not yet
reconcile phase started
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11 reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile failed

brute force -add owner but first try Organization Administrator (for policy creation) like the core-landing-zone package

gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/resourcemanager.organizationAdmin --condition=None --quiet  > /dev/null 1>&1

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/resourcemanager.organizationAdmin --condition=None --quiet  > /dev/null 1>&1
Updated IAM policy for organization [459065442144].

be patient took 5 min
  Warning  UpdateFailed  3m36s (x7 over 5m48s)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Error applying IAM policy for organization "459065442144": Error setting IAM policy for organization "459065442144": googleapi: Error 403: The caller does not have permission, forbidden
  Normal   UpToDate      90s                    iampolicymember-controller  The resource is up to date

  Warning  UpdateFailed  3m36s (x7 over 5m48s)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Error applying IAM policy for organization "459065442144": Error setting IAM policy for organization "459065442144": googleapi: Error 403: The caller does not have permission, forbidden
  Normal   UpToDate      90s                    iampolicymember-controller  The resource is up to date

specifically

iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   6m31s   True    UpToDate   2m12s

full network-sa service account roles

Access Context Manager Admin
Compute Shared VPC Admin

MISSING...
Organization Administrator
Service Account Admin

optional
Organization Role Administrator

raised https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/603

obriensystems commented 8 months ago

moving on to remaining issues update: roles/compute.instanceAdmin required on networking-sa to fix hub-mgmt-data-disk creation permissions error in hub-env

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking 
NAME                                                           AGE   READY   STATUS         STATUS AGE
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk   46m   False   UpdateFailed   46m

NAME                                                                       AGE    READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     176m   False   DependencyNotFound   176m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   176m   False   DependencyNotFound   176m

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance -n networking

  Warning  DependencyNotFound  3m10s (x18 over 177m)  computeinstance-controller  reference ComputeDisk networking/hub-fgt-primary-log-disk is not found

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance -n networking

  Warning  DependencyNotFound  6m58s (x20 over 179m)  computeinstance-controller  reference ComputeDisk networking/hub-fgt-secondary-log-disk is not found

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl describe computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk  -n networking

  Warning  UpdateFailed  50s (x30 over 49m)  computedisk-controller  Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing ComputeDisk "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/disks/mgmt-data-disk": googleapi: Error 403: Required 'compute.disks.get' permission for 'projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/disks/mgmt-data-disk', forbidden

checking permissions

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud projects get-iam-policy xxdmu-admin1-hub-oi11
bindings:
- members:
  - serviceAccount:service-851414103698@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:851414103698-compute@developer.gserviceaccount.com
  - serviceAccount:851414103698@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:projects-sa@kcc-oi-3552.iam.gserviceaccount.com
  role: roles/owner

adjusted project.yaml back to (from bottom 3 commented)
    cnrm.cloud.google.com/auto-create-network: "false"
    #config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure
    #internal.kpt.dev/upstream-identifier: 'resourcemanager.cnrm.cloud.google.com|Project|projects|hub-project-id'
    cnrm.cloud.google.com/blueprint: 'kpt-pkg-fn-live'

add to networking-sa 
roles/compute.instanceAdmin

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/compute.instanceAdmin --condition=None --quiet  > /dev/null 1>&1
Updated IAM policy for organization [459065442144].

working
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking 
NAME                                                           AGE   READY   STATUS     STATUS AGE
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk   61m   True    UpToDate   67s

NAME                                                                       AGE     READY   STATUS               STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     3h12m   False   DependencyNotFound   3h12m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   3h12m   False   DependencyNotFound   3h12m

verified
  Warning  UpdateFailed  4m47s (x35 over 63m)  computedisk-controller  Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing ComputeDisk "projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/disks/mgmt-data-disk": googleapi: Error 403: Required 'compute.disks.get' permission for 'projects/xxdmu-admin1-hub-oi11/zones/northamerica-northeast1-a/disks/mgmt-data-disk', forbidden
  Normal   Updating      2m46s                 computedisk-controller  Update in progress
  Normal   UpToDate      2m33s                 computedisk-controller  The resource is up to date

obriensystems commented 8 months ago

raised for above https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/604

attempting re-render

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11 apply successful
apply phase finished
reconcile phase started
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11 reconcile successful

obriensystems commented 8 months ago

missing a lot of the networking stack checked with kpt alpha live plan hub-env and I see the yaml plan contains them but not in gke objects - does not make it to rendering

ichael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt alpha live plan hub-env ...

compute.cnrm.cloud.google.com/ComputeInstance networking/hub-management-instance apiVersion: compute.cnrm.cloud.google.com/v1beta1 kind: ComputeInstance metadata: annotations: cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live cnrm.cloud.google.com/project-id: xxdmu-admin1-hub-oi11 cnrm.cloud.google.com/state-into-spec: absent config.k8s.io/owning-inventory: b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809 config.kubernetes.io/depends-on: compute.cnrm.cloud.google.com/namespaces/networking/ComputeSubnetwork/hub-nane1-mgmt-rz-snet config.kubernetes.io/path: fortigate/management-vm/management-vm.yaml internal.config.kubernetes.io/path: fortigate/management-vm/management-vm.yaml internal.kpt.dev/upstream-identifier: compute.cnrm.cloud.google.com|ComputeInstance|networking|hub-management-instance name: hub-management-instance namespace: networking spec: attachedDisk:
- sourceDiskRef: name: hub-mgmt-data-disk bootDisk: autoDelete: true deviceName: persistent-disk-0 initializeParams: size: 50 sourceImageRef: external: windows-cloud/windows-2022 type: pd-standard description: Hub Management VM machineType: n2-standard-2 networkInterface:
- name: nic0 networkRef: name: hub-global-mgmt-vpc subnetworkRef: name: hub-nane1-mgmt-rz-snet resourceID: management-instance scheduling: automaticRestart: true onHostMaintenance: MIGRATE serviceAccount: scopes:
  - cloud-platform serviceAccountRef: name: hub-managementvm-sa shieldedInstanceConfig: enableIntegrityMonitoring: true enableSecureBoot: true enableVtpm: true zone: northamerica-northeast1-a
compute.cnrm.cloud.google.com/ComputeRoute networking/hub-internal-vpc-internet-egress-route apiVersion: compute.cnrm.cloud.google.com/v1beta1 kind: ComputeRoute metadata: annotations: cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live cnrm.cloud.google.com/project-id: xxdmu-admin1-hub-oi11 config.k8s.io/owning-inventory: b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809 config.kubernetes.io/depends-on: compute.cnrm.cloud.google.com/namespaces/networking/ComputeForwardingRule/hub-ilb-fwdrule config.kubernetes.io/path: fortigate/route.yaml internal.config.kubernetes.io/path: fortigate/route.yaml internal.kpt.dev/upstream-identifier: compute.cnrm.cloud.google.com|ComputeRoute|networking|hub-internal-vpc-internet-egress-route name: hub-internal-vpc-internet-egress-route namespace: networking spec: description: route to the internet destRange: 0.0.0.0/0 networkRef: name: hub-global-internal-vpc nextHopILBRef: name: hub-ilb-fwdrule priority: 100 resourceID: internal-internet-egress-route
compute.cnrm.cloud.google.com/ComputeRouterNAT networking/hub-nane1-external-nat apiVersion: compute.cnrm.cloud.google.com/v1beta1 kind: ComputeRouterNAT metadata: annotations: cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live cnrm.cloud.google.com/project-id: xxdmu-admin1-hub-oi11 config.k8s.io/owning-inventory: b05327c7f4795bc3d0a64465223d68bb15ac3e45-1698077274333035809 config.kubernetes.io/depends-on: compute.cnrm.cloud.google.com/namespaces/networking/ComputeRouter/hub-nane1-external-router config.kubernetes.io/path: network/nat.yaml internal.config.kubernetes.io/path: network/nat.yaml internal.kpt.dev/upstream-identifier: compute.cnrm.cloud.google.com|ComputeRouterNAT|networking|hub-nane1-external-nat name: hub-nane1-external-nat namespace: networking spec: natIpAllocateOption: AUTO_ONLY region: northamerica-northeast1 resourceID: nane1-external-nat routerRef: name: hub-nane1-external-router sourceSubnetworkIpRangesToNat: ALL_SUBNETWORKS_ALL_IP_RANGES

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env --reconcile-timeout=5m --output=table networking ComputeInstance/hub-fgt-primary-instance Skipped InProgress Ready 4h reference ComputeDisk networking/hub-fgt networking ComputeInstance/hub-fgt-secondary-instan Skipped InProgress Ready 4h reference ComputeDisk networking/hub-fgt networking ComputeInstance/hub-management-instance Skipped Unknown - -
networking ComputeInstanceGroup/hub-fgt-primary-umi Skipped Unknown - -
networking ComputeInstanceGroup/hub-fgt-secondary-u Skipped Unknown - -
networking ComputeNetwork/hub-global-external-vpc Skipped Unknown - -
networking ComputeNetwork/hub-global-internal-vpc Skipped Unknown - -
networking ComputeNetwork/hub-global-mgmt-vpc Skipped Unknown - -
networking ComputeNetwork/hub-global-transit-vpc Skipped Unknown - -
networking ComputeRoute/hub-external-vpc-internet-e Skipped Unknown - -
networking ComputeRoute/hub-internal-vpc-internet-e Skipped Unknown - -
networking ComputeRouter/hub-nane1-external-router Skipped Unknown - -
networking ComputeRouterNAT/hub-nane1-external-nat Skipped Unknown - -
networking ComputeSubnetwork/hub-nane1-external-paz Skipped Unknown - -
networking ComputeSubnetwork/hub-nane1-internal-paz Skipped Unknown - -
networking ComputeSubnetwork/hub-nane1-mgmt-rz-snet Skipped Unknown - -
networking ComputeSubnetwork/hub-nane1-transit-paz- Skipped Unknown - -
networking ComputeTargetPool/hub-elb-pool Skipped Unknown - -
networking DNSPolicy/hub-external-logging-dnspolicy Skipped Unknown - -
networking DNSPolicy/hub-internal-logging-dnspolicy Skipped Unknown - -
networking DNSPolicy/hub-mgmt-logging-dnspolicy Skipped Unknown - -
networking DNSPolicy/hub-transit-logging-dnspolicy Skipped Unknown - -
networking IAMPolicyMember/fortigatesdn-sa-fortigat Skipped Current Ready 68m Resource is Current
networking IAMPolicyMember/hub-admin-serviceaccount Skipped Current Ready 4h Resource is Current
networking IAMServiceAccount/hub-fortigatesdn-sa Skipped Current Ready 2h Resource is Current
networking IAMServiceAccount/hub-managementvm-sa Skipped Current Ready 4h Resource is Current
policies ResourceManagerPolicy/compute-disable-se Skipped Unknown - -
policies ResourceManagerPolicy/compute-require-sh Skipped Unknown - -
policies ResourceManagerPolicy/compute-restrict-l Skipped Unknown - -
policies ResourceManagerPolicy/compute-restrict-v Skipped Unknown - -
policies ResourceManagerPolicy/compute-trusted-im Skipped Unknown - -
policies ResourceManagerPolicy/compute-vm-can-ip- Skipped Unknown - -
policies ResourceManagerPolicy/compute-vm-externa Skipped Unknown - -
projects Project/xxdmu-admin1-hub-oi11 Skipped Current Ready 4h Resource is Current
projects Service/xxdmu-admin1-hub-oi11-compute Skipped Unknown - -
projects Service/xxdmu-admin1-hub-oi11-dns Skipped Unknown - -

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env installing inventory ResourceGroup CRD. inventory update started inventory update finished apply phase started computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11 apply skipped: inventory policy prevented actuation (strategy: Apply, status: NoMatch, policy: MustMatch) apply phase finished reconcile phase started computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile skipped computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile skipped computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile skipped iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile skipped iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile skipped project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11 reconcile skipped reconcile phase finished apply phase started computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig apply skipped: dependency apply actuation skipped: networking_hub-fgt-primary-instance_compute.cnrm.cloud.google.com_ComputeInstance computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig apply skipped: dependency apply actuation skipped: networking_hub-fgt-secondary-instance_compute.cnrm.cloud.google.com_ComputeInstance computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool apply skipped: dependency apply actuation skipped: networking_hub-fgt-primary-instance_compute.cnrm.cloud.google.com_ComputeInstance iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11-compute apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11-dns apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11_resourcemanager.cnrm.cloud.google.com_Project apply phase finished reconcile phase started computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig reconcile skipped computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig reconcile skipped computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool reconcile skipped iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile skipped iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile skipped resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile skipped service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11-compute reconcile skipped service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi11-dns reconcile skipped reconcile phase finished apply phase started computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes apply skipped: dependency apply actuation skipped: networking_hub-fgt-primary-umig_compute.cnrm.cloud.google.com_ComputeInstanceGroup computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply skipped: dependency apply actuation skipped: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply skipped: dependency apply actuation skipped: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply skipped: dependency apply actuation skipped: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply skipped: dependency apply actuation skipped: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11-compute_serviceusage.cnrm.cloud.google.com_Service computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11-compute_serviceusage.cnrm.cloud.google.com_Service computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11-compute_serviceusage.cnrm.cloud.google.com_Service computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply skipped: dependency apply actuation skipped: projects_xxdmu-admin1-hub-oi11-compute_serviceusage.cnrm.cloud.google.com_Service apply phase finished reconcile phase started computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes reconcile skipped computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile skipped computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile skipped computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile skipped computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile skipped computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile skipped computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile skipped computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile skipped computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile skipped reconcile phase finished apply phase started computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-external-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-mgmt-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-internal-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-external-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-mgmt-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-internal-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr apply skipped: dependency apply actuation skipped: networking_hub-global-mgmt-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule apply skipped: dependency apply actuation skipped: networking_hub-ilb-bes_compute.cnrm.cloud.google.com_ComputeBackendService computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule apply skipped: dependency apply actuation skipped: networking_hub-ilb-bes_compute.cnrm.cloud.google.com_ComputeBackendService computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route apply skipped: dependency apply actuation skipped: networking_hub-global-external-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router apply skipped: dependency apply actuation skipped: networking_hub-global-external-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet apply skipped: dependency apply actuation skipped: networking_hub-global-external-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet apply skipped: dependency apply actuation skipped: networking_hub-global-internal-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet apply skipped: dependency apply actuation skipped: networking_hub-global-mgmt-vpc_compute.cnrm.cloud.google.com_ComputeNetwork computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet apply skipped: dependency apply actuation skipped: networking_hub-global-transit-vpc_compute.cnrm.cloud.google.com_ComputeNetwork dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy apply skipped: dependency apply actuation skipped: networking_hub-global-external-vpc_compute.cnrm.cloud.google.com_ComputeNetwork dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy apply skipped: dependency apply actuation skipped: networking_hub-global-internal-vpc_compute.cnrm.cloud.google.com_ComputeNetwork dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy apply skipped: dependency apply actuation skipped: networking_hub-global-mgmt-vpc_compute.cnrm.cloud.google.com_ComputeNetwork dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy apply skipped: dependency apply actuation skipped: networking_hub-global-transit-vpc_compute.cnrm.cloud.google.com_ComputeNetwork apply phase finished reconcile phase started computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile skipped computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile skipped computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile skipped computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile skipped computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile skipped computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile skipped computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr reconcile skipped computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule reconcile skipped computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule reconcile skipped computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile skipped computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile skipped computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile skipped computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile skipped computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile skipped computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile skipped dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile skipped dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile skipped dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile skipped dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile skipped reconcile phase finished apply phase started computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-external-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-internal-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-mgmt-rz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-transit-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-external-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-internal-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-mgmt-rz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-transit-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-internal-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address apply skipped: dependency apply actuation skipped: networking_hub-nane1-internal-paz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeinstance.compute.cnrm.cloud.google.com/hub-management-instance apply skipped: dependency apply actuation skipped: networking_hub-nane1-mgmt-rz-snet_compute.cnrm.cloud.google.com_ComputeSubnetwork computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route apply skipped: dependency apply actuation skipped: networking_hub-ilb-fwdrule_compute.cnrm.cloud.google.com_ComputeForwardingRule computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat apply skipped: dependency apply actuation skipped: networking_hub-nane1-external-router_compute.cnrm.cloud.google.com_ComputeRouter apply phase finished reconcile phase started computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address reconcile skipped computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address reconcile skipped computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile skipped computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route reconcile skipped computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat reconcile skipped reconcile phase finished inventory update started inventory update finished apply result: 67 attempted, 0 successful, 67 skipped, 0 failed reconcile result: 67 attempted, 0 successful, 67 skipped, 0 failed, 0 timed out michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$

obriensystems commented 8 months ago

I accidentally kpt init'd the env - all hub-env resources are skipped I will try a kubectl delete gcp --all after I destroy the core-landing-zone and restart - likely from a clean project

obriensystems commented 8 months ago

in landing.systems cloned repo removed all depends-on rendering in place in the repo

hanges not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   solutions/project/hub-env/fortigate/address.yaml
        modified:   solutions/project/hub-env/fortigate/disk.yaml
        modified:   solutions/project/hub-env/fortigate/elb.yaml
        modified:   solutions/project/hub-env/fortigate/firewall.yaml
        modified:   solutions/project/hub-env/fortigate/fortigate-ap-primary.yaml
        modified:   solutions/project/hub-env/fortigate/fortigate-ap-secondary.yaml
        modified:   solutions/project/hub-env/fortigate/ilb.yaml
        modified:   solutions/project/hub-env/fortigate/management-vm/disk.yaml
        modified:   solutions/project/hub-env/fortigate/management-vm/firewall.yaml
        modified:   solutions/project/hub-env/fortigate/management-vm/management-vm.yaml
        modified:   solutions/project/hub-env/fortigate/management-vm/service-account.yaml
        modified:   solutions/project/hub-env/fortigate/route.yaml
        modified:   solutions/project/hub-env/fortigate/service-account.yaml
        modified:   solutions/project/hub-env/fortigate/umig.yaml
        modified:   solutions/project/hub-env/network/dns.yaml
        modified:   solutions/project/hub-env/network/nat.yaml
        modified:   solutions/project/hub-env/network/route.yaml
        modified:   solutions/project/hub-env/network/subnet.yaml
        modified:   solutions/project/hub-env/network/vpc.yaml
        modified:   solutions/project/hub-env/setters.yaml
        modified:   solutions/setup.sh
        modified:   solutions/vars.sh

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ cd project/
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ ls
hub-env  project-experimentation  spoke-unclass-env
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kpt live init hub-env --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls)$ cd project/
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ ls
hub-env  project-experimentation  spoke-unclass-env
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kpt live init hub-env --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

forgot depends on in 
        modified:   hub-env/project-iam.yaml
        modified:   hub-env/project.yaml
        modified:   hub-env/services.yaml

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kpt fn render hub-env --truncate-output=false
Package "hub-env": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 500ms
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "59485982875"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.bootDisk.initializeParams.sourceImageRef.external: set field value to "projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license"
    [info] spec.metadata[1].value: set field value to "LICENSE\n"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.bootDisk.initializeParams.sourceImageRef.external: set field value to "projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license"
    [info] spec.metadata[1].value: set field value to "LICENSE\n"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "user:root@landing.systems"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "serviceAccount:fortigatesdn-sa@xxdmu-admin1-ls.iam.gserviceaccount.com"
    [info] spec.role: set field value to "organizations/59485982875/roles/FortigateSdnViewer"
    [info] spec.resourceRef.external: set field value to "59485982875"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.listPolicy.allow.values: set field value to "- under:organizations/59485982875\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/xxdmu-admin1-ls/zones/northamerica-northeast1-a/instances/fgt-primary-instance\"\n- \"projects/xxdmu-admin1-ls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance\"\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] spec.listPolicy.allow.values: set field value to "- \"projects/xxdmu-admin1-ls/zones/northamerica-northeast1-a/instances/fgt-primary-instance\"\n- \"projects/xxdmu-admin1-ls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance\"\n"
    [info] spec.projectRef.external: set field value to "xxdmu-admin1-ls"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "user:root@landing.systems"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
    [info] spec.resourceRef.external: set field value to "xxdmu-admin1-ls"
    [info] spec.member: set field value to "user:root@landing.systems"
    [info] metadata.name: set field value to "xxdmu-admin1-ls"
    [info] spec.name: set field value to "xxdmu-admin1-ls"
    [info] spec.billingAccountRef.external: set field value to "01E6E8-A42E99-D21FF3"
    [info] spec.folderRef.name: set field value to "services-infrastructure"
    [info] metadata.name: set field value to "xxdmu-admin1-ls-compute"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-ls"
    [info] metadata.name: set field value to "xxdmu-admin1-ls-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "xxdmu-admin1-ls"
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 400ms
  Results:
    [info]: no matches

adding to networking-sa

running with
apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  # Organization ID see usage in the custom-role.yaml
  org-id: "59485982875"
  # Billing Account ID to be associated with this project
  project-billing-id: "01E6E8-A42E99-D21FF3"
  # GCP folder to use as parent to this project, lowercase K8S resource name
  project-parent-folder: services-infrastructure
  # Naming Convention for project-id : <tenant-code><environment-code>m<data-classification>-<project-owner>-<user defined string>
  # Max 30 characters
  hub-project-id: xxdmu-admin1-ls
  # https://cloud.google.com/iap/docs/using-tcp-forwarding
  # Organization ID see usage in the custom-role.yaml
  management-project-id: "xxdmu-admin1-ls"
  # keep config-control as the default
  management-namespace: config-control
  # Identity that should be allowed to access the management VM using IAP TCP forwarding
  hub-admin: user:root@landing.systems
  #################
  # Org Policies
  #######
  # This list constraint defines the set of VPC networks
  # that are allowed to be peered with the VPC networks belonging to this project, see YAML file for more info:
  # org-policies/exceptions/compute-restrict-vpc-peering-except-hub-project.yaml
  # this setting MUST be changed to include the ORG ID
  project-allowed-restrict-vpc-peering: |
    - under:organizations/59485982875
  # This list constraint defines the set of Compute Engine VM instances that are allowed to use external IP addresses, see YAML file for more info:
  # org-policies/exceptions/compute-vm-external-ip-access-except-hub-project.yaml
  # this setting MUST be changed to include the hub project ID
  project-allowed-vm-external-ip-access: |
    - "projects/xxdmu-admin1-ls/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
    - "projects/xxdmu-admin1-ls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
  # This list constraint defines the set of VM instances that can enable IP forwarding., see YAML file for more info:
  # org-policies/exceptions/compute-vm-can-ip-forward-except-hub-project.yaml
  # this setting MUST be changed to include the hub project ID
  project-allowed-vm-can-ip-forward: |
    - "projects/xxdmu-admin1-ls/zones/northamerica-northeast1-a/instances/fgt-primary-instance"
    - "projects/xxdmu-admin1-ls/zones/northamerica-northeast1-b/instances/fgt-secondary-instance"
  #################
  # Fortigate
  #################
  # The Fortigate admin password cannot be defined in the setters.yaml file at the moment.
  # Until this is fixed, you will need to set it in the search-replace-config.yaml file.
  # fgt-admin-password: CHANGE_IN_search-replace-config.yaml
  #######
  # Primary
  # Having disctinct images allows one to use a Licensed Fortigate for the primary and a Pay-as-you-Go license for the secondary
  # and run the secondary just a couple of minutes each day for synching purposes thus obtaining an affordable cold standby.
  fgt-primary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
  # replace the word LICENSE below with the actual license value. Not required if using the Pay-as-you-Go image.
  fgt-primary-license: |
    LICENSE
  #######
  # Secondary
  fgt-secondary-image: projects/fortigcp-project-001/global/images/fortinet-fgtondemand-724-20230201-001-w-license
  # replace the word LICENSE below with the actual license value. Not required if using the Pay-as-you-Go image.
  fgt-secondary-license: |
    LICENSE

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kpt live apply hub-env

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address apply successful
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes apply successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply successful
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr apply successful
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule apply successful
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule apply successful
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply successful
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance apply successful
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig apply successful
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route apply successful
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route apply successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router apply successful
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet apply successful
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-ls apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns apply successful
apply phase finished
reconcile phase started
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address reconcile pending
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address reconcile pending
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile pending
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule reconcile pending
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule reconcile pending
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile pending
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile pending
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig reconcile pending
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile pending
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile pending
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route reconcile pending
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile pending
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile pending
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-ls reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile failed
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile failed
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns reconcile failed
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile failed
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile failed
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile failed
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile failed
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful

computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile failed
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile pending
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile pending
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile failed
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns reconcile successful

vpcs up


computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile failed
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile failed
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile pending
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile failed
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile pending
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile failed
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile failed
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile pending
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile pending
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile successful
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile successful
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile failed
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile pending
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile failed
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile failed
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile failed

4vpcs up fixing sa s

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp | grep False
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         5m51s   False   UpdateFailed   5m51s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          5m51s   False   UpdateFailed   5m51s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           5m51s   False   UpdateFailed   5m50s

  Warning  UpdateFailed  6m20s (x2 over 6m23s)  iampolicymember-controller  Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "project \"xxdmu-admin1-ls\"" with IAM Member: Role "roles/compute.instanceAdmin.v1" Member "serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com": Error retrieving IAM policy for project "xxdmu-admin1-ls": googleapi: Error 403: The caller does not have permission, forbidden
  Warning  UpdateFailed  92s (x6 over 5m58s)    iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com for project "xxdmu-admin1-ls"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1 serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com for project \"xxdmu-admin1-ls\"" both failed. Final error: Error applying IAM policy for project "xxdmu-admin1-ls": Error setting IAM policy for project "xxdmu-admin1-ls": googleapi: Error 400: Service account networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com does not exist., badRequest
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions

checking networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n networking
NAME                                                                             AGE     READY   STATUS     STATUS AGE
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address         9m49s   True    UpToDate   6m36s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address         9m49s   True    UpToDate   6m45s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address        9m48s   True    UpToDate   6m46s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address     9m48s   True    UpToDate   6m36s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address       9m48s   True    UpToDate   6m36s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address       9m48s   True    UpToDate   6m46s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address      9m48s   True    UpToDate   6m46s
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address   9m47s   True    UpToDate   6m36s
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address                     9m47s   True    UpToDate   6m45s
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address               9m47s   True    UpToDate   6m46s

NAME                                                              AGE     READY   STATUS               STATUS AGE
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes   9m47s   False   DependencyNotReady   9m47s

NAME                                                                   AGE     READY   STATUS     STATUS AGE
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk     9m47s   True    UpToDate   7m25s
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk   9m46s   True    UpToDate   7m25s
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk           9m46s   True    UpToDate   7m26s

NAME                                                                                         AGE     READY   STATUS     STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                         9m46s   True    UpToDate   6m48s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                    9m46s   True    UpToDate   6m57s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr             9m45s   True    UpToDate   6m58s
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr   9m45s   True    UpToDate   6m48s
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr          9m45s   True    UpToDate   6m57s
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr   9m45s   True    UpToDate   6m57s

NAME                                                                        AGE     READY   STATUS               STATUS AGE
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule         9m44s   False   DependencyNotReady   9m44s
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule   9m43s   False   DependencyNotReady   9m43s

NAME                                                                AGE     READY   STATUS     STATUS AGE
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc   9m42s   True    UpToDate   7m22s

NAME                                                                        AGE     READY   STATUS     STATUS AGE
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc   9m43s   True    UpToDate   7m22s

NAME                                                                        AGE     READY   STATUS               STATUS AGE
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     9m41s   False   DependencyNotReady   9m41s
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   9m41s   False   DependencyNotReady   9m41s

NAME                                                                       AGE     READY   STATUS         STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     9m42s   False   UpdateFailed   9m42s
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   9m42s   False   UpdateFailed   9m42s
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      9m42s   False   UpdateFailed   9m41s

NAME                                                                   AGE     READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   9m42s   True    UpToDate   109s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   9m42s   True    UpToDate   7m10s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       9m41s   True    UpToDate   7m10s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    9m41s   True    UpToDate   7m

NAME                                                                    AGE     READY   STATUS     STATUS AGE
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat   9m40s   True    UpToDate   6m38s

NAME                                                                    AGE     READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   9m40s   True    UpToDate   6m49s

NAME                                                                                AGE     READY   STATUS               STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   9m41s   True    UpToDate             6m49s
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route   9m41s   False   DependencyNotReady   9m41s

NAME                                                                          AGE     READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   9m40s   True    UpToDate   6m49s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   9m40s   True    UpToDate   6m58s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        9m39s   True    UpToDate   6m59s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    9m39s   True    UpToDate   6m48s

NAME                                                           AGE     READY   STATUS               STATUS AGE
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool   9m40s   False   DependencyNotReady   9m39s

NAME                                                                 AGE     READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   9m39s   True    UpToDate   6m54s
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   9m39s   True    UpToDate   7m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       9m39s   True    UpToDate   7m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    9m39s   True    UpToDate   6m54s

NAME                                                                                            AGE     READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   9m38s   False   UpdateFailed   9m37s
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              9m38s   True    UpToDate       9m26s

NAME                                                              AGE     READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   9m38s   True    UpToDate   9m31s
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   9m38s   True    UpToDate   9m31s

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n networking | grep False
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes   10m   False   DependencyNotReady   10m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule         10m   False   DependencyNotReady   10m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule   10m   False   DependencyNotReady   10m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     10m   False   DependencyNotReady   10m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   10m   False   DependencyNotReady   10m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     10m   False   UpdateFailed   10m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   10m   False   UpdateFailed   10m
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      10m   False   Updating       10m
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route   10m   False   DependencyNotReady   10m
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool   10m   False   DependencyNotReady   10m
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   10m   False   UpdateFailed   10m

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n networking | grep False
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes   10m   False   DependencyNotReady   10m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule         10m   False   DependencyNotReady   10m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule   10m   False   DependencyNotReady   10m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     10m   False   DependencyNotReady   10m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   10m   False   DependencyNotReady   10m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     10m   False   UpdateFailed   10m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   10m   False   UpdateFailed   10m
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      10m   False   Updating       10m
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route   10m   False   DependencyNotReady   10m
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool   10m   False   DependencyNotReady   10m
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   10m   False   UpdateFailed   10m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp | grep False
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         11m   False   UpdateFailed   11m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          11m   False   UpdateFailed   11m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           11m   False   UpdateFailed   11m

  Warning  UpdateFailed  34s (x14 over 16m)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Error applying IAM policy for organization "59485982875": Error setting IAM policy for organization "59485982875": googleapi: Error 400: Role (organizations/59485982875/roles/FortigateSdnViewer) does not exist in the resource's hierarchy., badRequest
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions -n networking

checking custom roles
didnt turn off depends on on custom roles

 name: hub-fortigatesdnreader-role

do another live apply
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful

fixed

  Warning  UpdateFailed  3m16s (x17 over 25m)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Error applying IAM policy for organization "59485982875": Error setting IAM policy for organization "59485982875": googleapi: Error 400: Role (organizations/59485982875/roles/FortigateSdnViewer) does not exist in the resource's hierarchy., badRequest
  Normal   UpToDate      71s                   iampolicymember-controller  The resource is up to date

vms coming up

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n networking
NAME                                                                             AGE   READY   STATUS     STATUS AGE
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address         26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address         26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address        26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address     26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address       26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address       26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address      26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address   26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address                     26m   True    UpToDate   23m
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address               26m   True    UpToDate   23m

NAME                                                              AGE   READY   STATUS               STATUS AGE
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes   26m   False   DependencyNotReady   26m

NAME                                                                   AGE   READY   STATUS     STATUS AGE
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk     26m   True    UpToDate   24m
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk   26m   True    UpToDate   24m
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk           26m   True    UpToDate   24m

NAME                                                                                         AGE   READY   STATUS     STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                         26m   True    UpToDate   23m
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                    26m   True    UpToDate   23m
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr             26m   True    UpToDate   23m
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr   26m   True    UpToDate   23m
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr          26m   True    UpToDate   23m
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr   26m   True    UpToDate   23m

NAME                                                                        AGE   READY   STATUS               STATUS AGE
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule         26m   False   DependencyNotReady   26m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule   26m   False   DependencyNotReady   26m

NAME                                                                AGE   READY   STATUS     STATUS AGE
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc   26m   True    UpToDate   24m

NAME                                                                        AGE   READY   STATUS     STATUS AGE
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc   26m   True    UpToDate   24m

NAME                                                                        AGE   READY   STATUS               STATUS AGE
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     26m   False   DependencyNotReady   26m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   26m   False   DependencyNotReady   26m

NAME                                                                       AGE   READY   STATUS         STATUS AGE
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     26m   False   UpdateFailed   26m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   26m   False   UpdateFailed   26m
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      26m   False   UpdateFailed   26m

NAME                                                                   AGE   READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   26m   True    UpToDate   6m24s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   26m   True    UpToDate   4m38s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       26m   True    UpToDate   5m39s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    26m   True    UpToDate   10m

NAME                                                                    AGE   READY   STATUS     STATUS AGE
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat   26m   True    UpToDate   23m

NAME                                                                    AGE   READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   26m   True    UpToDate   23m

NAME                                                                                AGE   READY   STATUS               STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   26m   True    UpToDate             23m
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route   26m   False   DependencyNotReady   26m

NAME                                                                          AGE   READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   26m   True    UpToDate   23m
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   26m   True    UpToDate   23m
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        26m   True    UpToDate   23m
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    26m   True    UpToDate   23m

NAME                                                           AGE   READY   STATUS               STATUS AGE
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool   26m   False   DependencyNotReady   26m

NAME                                                                 AGE   READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   26m   True    UpToDate   23m
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   26m   True    UpToDate   23m
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       26m   True    UpToDate   23m
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    26m   True    UpToDate   23m

NAME                                                                                            AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions   26m   True    UpToDate   116s
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions              26m   True    UpToDate   26m

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa   26m   True    UpToDate   26m
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa   26m   True    UpToDate   26m

2 vms - but both are retries

obriensystems commented 8 months ago

working remaining

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n networking | grep  False
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes   35m   False   DependencyNotReady   35m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule         34m   False   DependencyNotReady   34m
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule   34m   False   DependencyNotReady   34m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     34m   False   DependencyNotReady   34m
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   34m   False   DependencyNotReady   34m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     34m   False   UpdateFailed   34m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   34m   False   UpdateFailed   34m
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      34m   False   UpdateFailed   34m
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route   34m   False   DependencyNotReady   34m
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool   34m   False   DependencyNotReady   34m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp  | grep  False
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         35m   False   UpdateFailed   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          35m   False   UpdateFailed   35m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           35m   False   UpdateFailed   35m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n projects | grep  False
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   44h   False   DependencyNotFound   44h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         44h   False   DependencyNotFound   44h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               44h   False   DependencyNotFound   44h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      44h   False   DependencyNotFound   44h
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n logging | grep  False
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n hierarchy | grep  False
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl get gcp -n policies | grep  False

sa is wrong
 networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com
should be
 networking-sa@kcc-boot-ls-8704.iam.gserviceaccount.com

 found it in project-iam.yaml
member: "serviceAccount:networking-sa@xxdmu-admin1-ls.iam.gserviceaccount.com" # kpt-set: serviceAccount:networking-sa@${management-project-id}.iam.gserviceaccount.com
should be
management-project-id:  kcc-boot-ls-8704

rendering and applying
had a space
  management-project-id: " kcc-boot-ls-8704"
  retrying

Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-computeinstanceadmin-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply failed: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"iam.cnrm.cloud.google.com/v1beta1\",\"kind\":\"IAMPolicyMember\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/blueprint\":\"kpt-fn-live\",\"cnrm.cloud.google.com/ignore-clusterless\":\"true\",\"cnrm.cloud.google.com/project-id\":\"xxdmu-admin1-ls\",\"config.k8s.io/owning-inventory\":\"29d822620e02c71cbdac3b006a8848a8a0b49e7d-1698143463597904995\"},\"name\":\"networking-sa-serviceaccountadmin-permissions\",\"namespace\":\"config-control\"},\"spec\":{\"member\":\"serviceAccount:networking-sa@kcc-boot-ls-8704.iam.gserviceaccount.com\",\"resourceRef\":{\"apiVersion\":\"resourcemanager.cnrm.cloud.google.com/v1beta1\",\"external\":\"xxdmu-admin1-ls\",\"kind\":\"Project\"},\"role\":\"roles/iam.serviceAccountAdmin\"}}\n"}},"spec":{"member":"serviceAccount:networking-sa@kcc-boot-ls-8704.iam.gserviceaccount.com"}}
to:
Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"

Resource: "iam.cnrm.cloud.google.com/v1beta1, Resource=iampolicymembers", GroupVersionKind: "iam.cnrm.cloud.google.com/v1beta1, Kind=IAMPolicyMember"
Name: "networking-sa-serviceaccountuser-permissions", Namespace: "config-control"
for: "project-iam.yaml": error when patching "project-iam.yaml": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: the IAMPolicyMember's spec is immutable
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-ls apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns apply successful
apply phase finished
reconcile phase started
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address reconcile successful
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile successful
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile successful
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule reconcile pending
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule reconcile pending
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile successful
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile pending
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig reconcile pending
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile successful
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route reconcile pending
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile successful
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile successful
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-ls reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-ls-dns reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile failed
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile failed
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile failed

  Warning  UpdateFailed  29m (x20 over 60m)    computeinstance-controller  Update call failed: error applying desired state: summary: Error waiting for instance to create: The user does not have access to service account 'fortigatesdn-sa@xxdmu-admin1-ls.iam.gserviceaccount.com'.  User: 'networking-sa@kcc-boot-ls-8704.iam.gserviceaccount.com'.  Ask a project owner to grant you the iam.serviceAccountUser role on the service account
  Normal   Updating      2m53s (x32 over 61m)  computeinstance-controller  Update in progress
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance  -n networking

reverting - the SA should be on the hub project
  #management-project-id: "kcc-boot-ls-8704"
  management-project-id: "xxdmu-admin1-ls"

5 min transit VPC good
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful

fmichaelobrien commented 8 months ago

reverting back to sa in cc project and Chris "update project in"

project-allowed-vm-can-ip-forward and project-allowed-vm-external-ip-access

triaging
  Warning  UpdateFailed  29m (x20 over 60m)    computeinstance-controller  Update call failed: error applying desired state: summary: Error waiting for instance to create: The user does not have access to service account 'fortigatesdn-sa@xxdmu-admin1-ls.iam.gserviceaccount.com'.  User: 'networking-sa@kcc-boot-ls-8704.iam.gserviceaccount.com'.  Ask a project owner to grant you the iam.serviceAccountUser role on the service account
  Normal   Updating      2m53s (x32 over 61m)  computeinstance-controller  Update in progress
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance  -n networking

obriensystems commented 8 months ago

A full double delete/create cycle and wait for the resources to delete 2 of the 3 VMs - up the remaining fortigate VM needs quota >8 - most likely due to repeated create/delete of projects - eventually we hit the 30d hoarding of quota - requesting an increase or switch vm type

including a temporary billing id typo caught by chris in projects.yaml where i had $c - thanks for the 2nd set of eyes

I have the diff in the repo including keeping all depends on and will correlate I didn't need the upstream annotation for projects.yaml - just the depends-on removal

last error one is quota for the 2nd VM - It may be the way deleted projects retain quota for 30 days - I don't have 7 instances up - checking
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls)$ kubectl describe computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance -n networking
Warning  UpdateFailed        4m9s (x4 over 4m44s)   computeinstance-controller  Update call failed: error applying desired state: summary: Error waiting for instance to create: Quota 'N2_CPUS' exceeded.  Limit: 8.0 in region northamerica-northeast1.
           metric name = compute.googleapis.com/n2_cpus
           limit name = N2-CPUS-per-project-region
           limit = 8
           dimensions = map[region:northamerica-northeast1]
  Normal   Updating  2m37s (x7 over 4m56s)  computeinstance-controller  Update in progress

obriensystems commented 8 months ago

Switching from payg to byod for the fortigate VMs

check instances with
kubectl get computeinstance -n networking

or 
kubectl get gcp -n networking | grep computeinstance

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls-8704)$ kubectl get computeinstance -n networking
NAME                         AGE    READY   STATUS         STATUS AGE
hub-fgt-primary-instance     3d1h   True    UpToDate       44m
hub-fgt-secondary-instance   3d1h   False   UpdateFailed   3d1h
hub-management-instance      3d1h   True    UpToDate       3d1h
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls-8704)$ kubectl get gcp -n networking | grep computeinstance
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     3d1h   True    UpToDate             3d1h
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   3d1h   False   DependencyNotReady   3d1h
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     3d1h   True    UpToDate       46m
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   3d1h   False   UpdateFailed   3d1h
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      3d1h   True    UpToDate       3d1h

determining whether single or multi line in the setters.yaml

remove the comments before/after the licens

switch billing- as I it switched back and I was charged an extr $50/d

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls-8704)$ kpt fn render hub-env --truncate-output=false
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls-8704)$ kpt live apply hub-env

installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address apply successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address apply successful
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes apply successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply successful
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr apply successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr apply successful
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule apply successful
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule apply successful
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply successful
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance apply successful
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance apply successful
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig apply successful
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route apply successful
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route apply successful
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router apply successful
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet apply successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet apply successful
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy apply successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy apply successful
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin4-ls-dns apply successful
apply phase finished
reconcile phase started
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-ext-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-int-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-mgmt-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-primary-transit-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-ext-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-int-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-mgmt-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-fgt-secondary-transit-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-address reconcile successful
computeaddress.compute.cnrm.cloud.google.com/hub-ilb-proxy-address reconcile successful
computebackendservice.compute.cnrm.cloud.google.com/hub-ilb-bes reconcile pending
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile successful
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile successful
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr reconcile successful
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr reconcile successful
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-fwdrule reconcile pending
computeforwardingrule.compute.cnrm.cloud.google.com/hub-ilb-proxy-fwdrule reconcile pending
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile successful
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile pending
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance reconcile successful
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig reconcile successful
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile successful
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route reconcile successful
computeroute.compute.cnrm.cloud.google.com/hub-internal-vpc-internet-egress-route reconcile pending
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router reconcile successful
computerouternat.compute.cnrm.cloud.google.com/hub-nane1-external-nat reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet reconcile successful
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet reconcile successful
computetargetpool.compute.cnrm.cloud.google.com/hub-elb-pool reconcile pending
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy reconcile successful
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/fortigatesdn-sa-fortigatesdnviewer-role-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-serviceaccountuser-permissions reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin4-ls-dns reconcile successful
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile failed
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance reconcile successful
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance reconcile failed
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin4-ls reconcile failed

w
>mVYiD_~#kgkM^Z

billing not updating

for ssh add

VM has a [firewall rule](https://cloud.google.com/iap/docs/using-tcp-forwarding#firewall) that allows TCP ingress traffic from the IP range 35.235.240.0/20, port: 22

obriensystems commented 8 months ago

Reduced machine size temporarily to n2-standard-2 (6 cores for 3 vms) wont work because of the nic count

  Warning  UpdateFailed  6s                    computeinstance-controller  Update call failed: error applying desired state: summary: Error creating instance: googleapi: Error 400: Invalid value for field 'resource.networkInterfaces': ''. Too many network interfaces. The maximum number of network interfaces allowed for this machine type is 2., invalid

trying
  machineType: e2-standard-2
for management and n2-standard-2 for fg vms

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions/project (kcc-boot-ls-8704)$ kubectl get gcp -n networking | grep computeinstance
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-primary-umig     3d6h   True    UpToDate   109s
computeinstancegroup.compute.cnrm.cloud.google.com/hub-fgt-secondary-umig   3d6h   True    UpToDate   7s
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     3d6h   True    UpToDate   2m34s
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   3d6h   True    UpToDate   7s
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      3d6h   True    UpToDate   26s

gui has not caught up

quota issue solved by using a different machine type for the management instance and hit refresh on the gce pane

obriensystems commented 7 months ago

Test GKE Enterprise switchover anthos.googleapis.com fix

root_@cloudshell:~ (kcc-boot-ls-8704)$ gcloud services enable anthos.googleapis.com
Operation "operations/acat.p2-145363557028-3b75c0e2-51fe-421c-9990-b2ba2ff136af" finished successfully.

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Monitoring-and-Logging#gke-enterprise-dashboard

obriensystems commented 7 months ago

merge main

michaelobrien@mbp7 pubsec-declarative-toolkit % git merge main
Merge made by the 'ort' strategy.
 .github/workflows/scorecards.yml                                                                           |   2 +-
 .release-please-manifest.json                                                                              |   2 +-
 docs/landing-zone-v2/README.md                                                                             |  71 ++++++++++++++++++++++++++--
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/Kptfile                                  |  18 +++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/README.md                                |  77 ++++++++++++++++++++++++++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-tier4-sa.yaml                        |  33 +++++++++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-users.yaml                           |  63 +++++++++++++++++++++++++
 examples/landing-zone-v2/configconnector/tier3/client-project-iam/setters.yaml                             |  46 ++++++++++++++++++
 examples/landing-zone-v2/configconnector/tier3/cloud-armor/security-policy.yaml                            |  57 ++++++++++++++++++----
 solutions/core-landing-zone/CHANGELOG.md                                                                   |   7 +++
 solutions/core-landing-zone/org/org-sink.yaml                                                              |   3 ++
 solutions/experimentation/core-landing-zone/README.md                                                      | 180 +++++++++++++++++++++++++++++++++++++---------------------------------
 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/monitoring/metrics-scope.yaml |  23 +++++++++
 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/services.yaml                 |  41 ++++++++++++++++
 solutions/experimentation/core-landing-zone/namespaces/config-management-monitoring.yaml                   |  81 ++++++++++++++++++++++++++++++++
 solutions/experimentation/core-landing-zone/namespaces/logging.yaml                                        |  35 ++++++++++++++
 16 files changed, 639 insertions(+), 100 deletions(-)
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/Kptfile
 create mode 100755 examples/landing-zone-v2/configconnector/tier3/client-project-iam/README.md
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-tier4-sa.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/iam-users.yaml
 create mode 100644 examples/landing-zone-v2/configconnector/tier3/client-project-iam/setters.yaml
 create mode 100644 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/monitoring/metrics-scope.yaml
 create mode 100644 solutions/experimentation/core-landing-zone/lz-folder/audits/logging-project/services.yaml
 create mode 100644 solutions/experimentation/core-landing-zone/namespaces/config-management-monitoring.yaml

fmichaelobrien commented 7 months ago

From #654

KCC_PROJECT_ID reset on run KCC GKE cluster only without LZ deploy

  SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
  echo "post GKE cluster create - applying 2 roles to org: ${ORG_ID} and project: ${KCC_PROJECT_ID} on the yakima gke service account to prep for kpt deployment: $SA_EMAIL"
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/resourcemanager.organizationAdmin --condition=None --quiet  > /dev/null 1>&1
  gcloud projects add-iam-policy-binding "${KCC_PROJECT_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer" --project "${KCC_PROJECT_ID}" --quiet  > /dev/null 1>&1
  # need service account admin for kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.organizationRoleAdmin --condition=None --quiet > /dev/null 1>&1
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet > /dev/null 1>&1
fi

line 88

  # set KCC project id for case where we initially create the KCC cluster without rerunning with passed in -p project_id
  KCC_PROJECT_ID=$CC_PROJECT_ID

fmichaelobrien commented 7 months ago

State: Hi, there is a tracking issue on bringing up the fortigates that details every workaround/fix (3 so far) involved in deploying hub-env on top of core-landing-zone over the weekend of Oct 20th in prep of posting the deployment steps for wed the 25th. Most of the changes in the gh446-hub branch were merged into main at that time. The hub-env package is still being adjusted to bring it up to a full prod state in that branch.

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/446#issuecomment-1771365186 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh446-hub

such as
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/612/files#diff-c0fda181f1f31975bd6d6ccb1d9a1bb827a8ff54dac4ac7757bc6de13c0303d2L37

  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
-  namespace: config-control # kpt-set: ${management-namespace}
+  namespace: networking

in the larger set of issues https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues?q=is%3Aissue+is%3Aopen+label%3Afortinet

There is WIP automation going into automating the hub-env setters.yaml in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L357 The script above is a combination of KCC cluster bootstrap (reuse or recreation of the GKE cluster is optional) and deployment of the clz and hub-env packages - but it is still in dev.

The yakima role associations are in both scripts in addition to the readme at https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L209 and https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh#L28

thank you /michael

obriensystems commented 7 months ago

Note: config-control namespace override

Currently retesting in #654 and #611 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigate/service-account.yaml#L37 needs https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/612/files#diff-0453fd24870bbd1f648b0d14dcbd3877e4eceafba5cf535c1b962ac63969a94fR22 on top of existing https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/609 for https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/607
```
project/hub-env/fortigate/service-account.yaml:37
```

kind: IAMPolicyMember metadata: name: fortigatesdn-sa-fortigatesdnviewer-role-permissions namespace: config-control # kpt-set: ${management-namespace}

via project/hub-env/setters.yaml:22

keep config-control as the default

management-namespace: config-control

obriensystems commented 7 months ago

generated kcc project_id propagation to the end in yakima/sa role additions retested in #654

0648
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ ./setup.sh -b kcc-oi -u ar -n true -c true -l false -h false -d false -j false
existing project: 
Date: Wed 06 Dec 2023 11:48:39 AM UTC
Timestamp: 1701863319
running with: -b kcc-oi -u ar -c true -l false -h false -r false -d false -p 
Updated property [core/project].
Switched back to boot project kcc-oi
Start: 1701863320
unique string: ar
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc
Creating project: kcc-oi-6475
CC_PROJECT_ID: kcc-oi-6475
BOOT_PROJECT_ID: kcc-oi
BILLING_ID: 014479-806359-2F5F85
ORG_ID: 459..44
applying roles to the super admin SUPER_ADMIN_EMAIL: michael@obrien.industries
Updated IAM policy for organization [4..44].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..4].
Updated IAM policy for organization [4..].
Updated IAM policy for organization [4.4]..
Updated IAM policy for organization [4..144].
Updated IAM policy for organization [459..44].
Creating KCC project: kcc-oi-6475 on folder: 38862..43
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-oi-6475].
Waiting for [operations/cp.5638443903817105010] to finish...done.                                                                                           
Enabling service [cloudapis.googleapis.com] on project [kcc-oi-6475]...
Operation "operations/acat.p2-993154031891-29201c86-a034-44cc-a146-92e3e696b676" finished successfully.
Updated property [core/project] to [kcc-oi-6475].
Updated property [core/project].
Enabling billing on account: 014..85
billingAccountName: billingAccounts/014..5
billingEnabled: true
name: projects/kcc-oi-6475/billingInfo
projectId: kcc-oi-6475
sleep 45 sec before enabling services
Enabling APIs
Operation "operations/acf.p2-993154031891-7d0764e3-2cd3-49e7-8fb3-102ebcc9c323" finished successfully.
Operation "operations/acat.p2-993154031891-d64f4422-74fd-48c8-a84b-c664d443bb03" finished successfully.
Operation "operations/acat.p2-993154031891-512f8af5-90e8-42e4-8ec0-5b6ad758cf31" finished successfully.
Operation "operations/acat.p2-993154031891-cf30917a-8316-439f-b3c4-67035ae22681" finished successfully.
Operation "operations/acat.p2-993154031891-de537f80-1838-463a-991e-5dfb9fbcd191" finished successfully.
Operation "operations/acat.p2-993154031891-fc32e1ef-6444-4b10-af5a-73a29e981b21" finished successfully.
name: organizations/459065442144/settings
storageLocation: northamerica-northeast1
Create VPC: kcc-ls-vpc
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-6475/global/networks/kcc-ls-vpc].
NAME: kcc-ls-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network kcc-ls-vpc --allow tcp:22,tcp:3389,icmp

Create subnet kcc-ls-sn off VPC: kcc-ls-vpc using 192.168.0.0/16 on region: northamerica-northeast1
Created [https://www.googleapis.com/compute/v1/projects/kcc-oi-6475/regions/northamerica-northeast1/subnetworks/kcc-ls-sn].
NAME: kcc-ls-sn
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
create default firewalls
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc on project kcc-oi-6475
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-6475/locations/northamerica-northeast1/operations/operation-1701863484478-60bd5f872d47b-3203efc2-fe59f81d] to complet
e...working        

Waiting for operation [projects/kcc-oi-6475/locations/northamerica-northeast1/operations/operation-1701863484478-60bd5f872d47b-3203efc2-fe59f81d] to complet
e...working.                                                                                                                                                
e...working.                                                                                                                                                
e...working..                                                                                                                                               
e...done.                                                                                                                                                   
Created instance [kcc].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
Cluster create time: 1107 sec
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
List Clusters:
NAME: kcc
LOCATION: northamerica-northeast1
STATE: RUNNING
post GKE cluster create - applying 2 roles to org: 459065442144 and project: kcc-oi-6475 on the yakima gke service account to prep for kpt deployment: service-993154031891@gcp-sa-yakima.iam.gserviceaccount.com
Updated IAM policy for organization [459065442144].
Updated IAM policy for project [kcc-oi-6475].
Updated IAM policy for organization [459065442144].
Updated IAM policy for organization [459065442144].
Total Duration: 1282 sec
Date: Wed 06 Dec 2023 12:10:02 PM UTC
Timestamp: 1701864602
Updated property [core/project].
Switched back to boot project kcc-oi
**** Done ****
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi)$

711

obriensystems commented 6 months ago

delete/recreate KCC GKE cluster - then re-acquire resources by id https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh#L524

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit (kcc-boot-ls-8704)$ gcloud anthos config controller delete --location northamerica-northeast1 kcc-oi4
You are about to delete instance [kcc-oi4]

Do you want to continue (Y/n)?  Y

Delete request issued for: [kcc-oi4]
Waiting for operation [projects/kcc-boot-ls-8704/locations/northamerica-northeast1/operations/operation-1703862526504-60da768a36c60-f3b99c97-6bd1b089] to complete...working..   

Deleted instance [kcc-oi4].

fmichaelobrien commented 5 months ago

move the partially completed kpt version script in 446 that completed the core-landing-zone and was mid way through hub-env

to #766

Add client-landing-zone and client-setup
merge periodic package automations into main in phases

fmichaelobrien commented 5 months ago

Restarting hub-env deployment existing deployment moved from oi to ls

obriensystems commented 4 months ago

all 4 core-landing-zone, client-setup, client-landing-zone and client-project-setup done/fixed

client-project-setup setters generation

data:
  org-id: "${ORG_ID}"  
  management-project-id: "${KCC_PROJECT_ID}"
  management-namespace: "${MANAGEMENT_NAMESPACE}"
  client-name: client-${PREFIX_CLIENT_SETUP}
  client-management-project-id: client-management-project-${PREFIX_CLIENT_SETUP}
  host-project-id: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}
  # see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/network/subnet.yaml#L26
  #allowed-nane1-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  #allowed-nane2-main-subnet: net-host-project-${PREFIX_CLIENT_LANDING_ZONE}-nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane1-main-subnet: nane1-standard-${CLIENT_CLASSIFICATION}-main-snet
  allowed-nane2-main-subnet: nane2-standard-${CLIENT_CLASSIFICATION}-main-snet
  project-id: client-project-${PREFIX_CLIENT_PROJECT_SETUP}
  project-billing-id: "${BILLING_ID}"
#  project-parent-folder: clients.client-${PREFIX_CLIENT_SETUP}.standard.applications-infrastructure.${CLIENT_PROJECT_PARENT_FOLDER}
  project-parent-folder: standard.applications.${CLIENT_PROJECT_PARENT_FOLDER}
  repo-url: git-repo-to-observe
  repo-branch: main
  tier3-repo-dir: csync/tier3/configcontroller/deploy/env
  tier4-repo-dir: csync/tier4/configcontroller/deploy/env
EOF

see #766 see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/798/commits/84afc6165bd4ccc68c94775e250c2eb174f8b00e

GoogleCloudPlatform / pubsec-declarative-toolkit

Example 258 fortigate perimeter package deploy procedure/verify for core lz unmanaged client #446

updates

78

168

166

177

207

573

Notes:

org states

kcc.landing.systems

573

obrien.industries

landing systems

new org

TODO

Known Issues Workarounds

deploying hub package

Known Issues Workarounds

Install the KCC cluster and minimal set of Landing Zone packages

Scenarios

Plan

gcp-tools

refresh repo with main 20231019

create dev branch

add setters.yaml changes

undeploy any lz packages up

deploy changes - clz package of 4

push to remote repo

write part 2 lz sh script

Delete cluster only

Recreate KCC cluster

deploy changes - clz package of 4

triage - folder errors

delete partial lz

adjust setters.yaml

Triage pod failures

need to be done manually

switch to automation

but it is set in clz member: "serviceAccount:networking-sa@kcc-kls-cluster3.iam.gserviceaccount.com" # kpt-set: serviceAccount:networking-sa@${management-project-id}.iam.gserviceaccount.com

check the kpt alpha live plan

Hold: remaining services on

kcc.landing.systems

obrien.industries

destination.loggingLogBucketRef

587

testing clean install via generated setters

run 2 obrien.industries

keep config-control as the default

move the partially completed kpt version script in 446 that completed the core-landing-zone and was mid way through hub-env