search

GoogleCloudPlatform / pubsec-declarative-toolkit

The GCP PubSec Declarative Toolkit is a collection of declarative solutions to help you on your Journey to Google Cloud. Solutions are designed using Config Connector and deployed using Config Controller.
Apache License 2.0
30 stars 27 forks source link

Verify gke cluster deletion / recreation and reacquire of deployed resources (at least core-landing-zone) - for FinOps or non-drift requirements #794

Open obriensystems opened 5 months ago

obriensystems commented 5 months ago

For usually the FinOps case a developer or client wishes to shutdown the GKE cluster and behave like we used to with terraform - only use the IaC intent engine during actual changes. For sparse use cases where GCP infrastructure is rarely modified and drift protection is not required - keep the GKE cluster off

We will use the script from #446 or #766 and document on the staging wiki from this issue

./setup.sh -b kcc-oi -u ar -n false -c false -l false -h false -d true -j true -p kcc-oi-6475

delete/reassociate
gcloud anthos config controller delete --location $REGION $CLUSTER --quiet

create
gcloud anthos config controller create $CLUSTER --location $REGION --network $NETWORK --subnet $SUBNET --master-ipv4-cidr-block="172.16.0.128/28" --full-management
gcloud anthos config controller get-credentials $CLUSTER --location $REGION
gcloud anthos config controller list
  SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
  echo "post GKE cluster create - applying 2 roles to org: ${ORG_ID} and project: ${KCC_PROJECT_ID} on the yakima gke service account to prep for kpt deployment: $SA_EMAIL"
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/resourcemanager.organizationAdmin --condition=None --quiet  > /dev/null 1>&1
  gcloud projects add-iam-policy-binding "${KCC_PROJECT_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer" --project "${KCC_PROJECT_ID}" --quiet  > /dev/null 1>&1
  # need service account admin for kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa
  # Warning  UpdateFailed  36s (x9 over 6m44s)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).
  ##roles/iam.serviceAccountCreator
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.organizationRoleAdmin --condition=None --quiet > /dev/null 1>&1
  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet > /dev/null 1>&1

see also #818 #817

obriensystems commented 4 months ago

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#delete-the-kcc-gke-custer

Verify single package re-acquire before the full 5 package lz

michael@cloudshell:~/kcc-oi-20231206 (kcc-oi-7970)$ ls
github  kpt
michael@cloudshell:~/kcc-oi-20231206 (kcc-oi-7970)$ ls kpt
_20240130  core-landing-zone
michael@cloudshell:~/kcc-oi-20231206 (kcc-oi-7970)$ cd github/pubsec-declarative-toolkit/solutions/
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$

Check deployment

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ kubectl get gcp -n config-control
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           16d   True    UpToDate   16d
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   16d   True    UpToDate   16d

NAME                                                                     AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa   16d   True    UpToDate   16d
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa          16d   True    UpToDate   16d
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa                 16d   True    UpToDate   16d
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa                   16d   True    UpToDate   16d
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa                16d   True    UpToDate   16d
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa                  16d   True    UpToDate   16d
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa                  16d   True    UpToDate   16d

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-metric-writer-permissions                   16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-kcc-oi-7970-permissions                    16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              16d   True    UpToDate   16d

NAME                                                                                              AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/config-mgmt-mon-default-sa-workload-identity-binding   16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding          16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding                 16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding                   16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding                16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding                  16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding                  16d   True    UpToDate   16d

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-accesscontextmanager   16d   True    UpToDate   16d
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-anthos                 16d   True    UpToDate   16d
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-cloudbilling           16d   True    UpToDate   16d
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-cloudresourcemanager   16d   True    UpToDate   16d
service.serviceusage.cnrm.cloud.google.com/kcc-oi-7970-serviceusage           16d   True    UpToDate   16d

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ kubectl get gcp -n projects
NAME                                                                                                                   AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               16d   True    UpToDate   16d
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      16d   True    UpToDate   16d

NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   16d   True    UpToDate   16d

NAME                                                                                                       AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-monitoring-admin-logging-project-oi0130-permissions   16d   True    UpToDate   16d
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-storageadmin-logging-project-oi0130-permissions       16d   True    UpToDate   16d

NAME                                                                   AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi0130       16d   True    UpToDate   16d
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi0130   16d   True    UpToDate   16d

NAME                                                                           AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/dns-project-oi0130-dns              16d   True    UpToDate   16d
service.serviceusage.cnrm.cloud.google.com/logging-project-oi0130-logging      16d   True    UpToDate   16d
service.serviceusage.cnrm.cloud.google.com/logging-project-oi0130-monitoring   16d   True    UpToDate   16d
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ kubectl get gcp -n networking
NAME                                                                                   AGE   READY   STATUS     STATUS AGE
dnsmanagedzone.dns.cnrm.cloud.google.com/dns-project-oi0130-standard-core-public-dns   16d   True    UpToDate   16d
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    16d   True    UpToDate   16d
folder.resourcemanager.cnrm.cloud.google.com/clients                   16d   True    UpToDate   16d
folder.resourcemanager.cnrm.cloud.google.com/services                  16d   True    UpToDate   16d
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   16d   True    UpToDate   16d
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ kubectl get gcp -n policies
NAME                                                                                                                 AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-guest-attribute-access                   16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-nested-virtualization                    16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access                       16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging                      16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-logging-except-kcc-oi-7970   16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-vpc-external-ipv6                        16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-os-login                                 16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm                              16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-kcc-oi-7970           16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage                         16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-oi-7970      16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types        16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-shared-vpc-lien-removal                 16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering                             16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpn-peer-ips                            16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-skip-default-network-creation                    16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects                           16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward                                16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access                            16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/essentialcontacts-allowed-contact-domains                16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/gcp-restrict-resource-locations                          16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-allowed-policy-member-domains                        16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-automatic-iam-grants-for-default-service-accounts    16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-audit-logging-exemption                      16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-creation                 16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/iam-disable-service-account-key-upload                   16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/sql-restrict-public-ip                                   16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention                         16d   True    UpToDate   16d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-uniform-bucket-level-access                      16d   True    UpToDate   16d
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ kubectl get gcp -n logging
NAME                                                                                                AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/logging-project-oi0130-data-access-sink                16d   True    UpToDate   16d
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-sink   16d   True    UpToDate   16d
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-data-access-logging-project-oi0130        16d   True    UpToDate   16d
logginglogsink.logging.cnrm.cloud.google.com/org-log-sink-security-logging-project-oi0130           16d   True    UpToDate   16d
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-infra-log-sink         16d   True    UpToDate   16d
logginglogsink.logging.cnrm.cloud.google.com/platform-and-component-services-log-sink               16d   True    UpToDate   16d

NAME                                                                                      AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-oi0130   16d   True    UpToDate   16d
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket                        16d   True    UpToDate   16d

NAME                                                                      AGE   READY   STATUS     STATUS AGE
monitoringmonitoredproject.monitoring.cnrm.cloud.google.com/kcc-oi-7970   16d   True    UpToDate   16d

NAME                                                                              AGE   READY   STATUS     STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/security-incident-log-bucket-oi0130   14d   True    UpToDate   14d
obriensystems commented 4 months ago

destroy gke cluster

Screenshot 2024-02-16 at 11 48 47 Screenshot 2024-02-16 at 11 50 35 Screenshot 2024-02-16 at 11 49 06 
michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ gcloud anthos config controller delete --location northamerica-northeast1 kcc  --quiet
Delete request issued for: [kcc]
Waiting for operation [projects/kcc-oi-7970/locations/northamerica-northeast1/operations/operation-1708103535888-61182d87f07de-28fac1d2-2f39769d] to complete...working... 

6 min
Deleted instance [kcc].
Screenshot 2024-02-16 at 12 23 35

wait 24+ hours and restart the cluster

fmichaelobrien commented 4 months ago

Check autopilot upgrades while the cluster was deleted

obriensystems commented 3 months ago

recreating cluster follow https://medium.com/@kasiarun/introduction-to-anthos-config-management-1a43917c26ae https://cloud.google.com/anthos-config-management/docs/config-sync-overview https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/README.md#clean-up

check nomos for config-controller https://cloud.google.com/anthos-config-management/docs/how-to/config-controller-migration#migrate-all

resources are in abandoned state https://cloud.google.com/anthos-config-management/docs/how-to/config-controller-setup#delete

follow https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh#L198

michael@cloudshell:~$ gcloud config set project kcc-oi-7970
Updated property [core/project].
michael@cloudshell:~ (kcc-oi-7970)$ cd kcc-oi-20231206/
michael@cloudshell:~/kcc-oi-20231206 (kcc-oi-7970)$ cd kpt/
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ NETWORK=kcc-vpc
SUBNET=kcc-sn
export REGION=northamerica-northeast1
CLUSTER=kcc
echo "Creating Anthos KCC autopilot cluster ${CLUSTER} in region ${REGION} in subnet ${SUBNET} off VPC ${NETWORK}"
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-sn off VPC kcc-vpc
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ 

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET" --master-ipv4-cidr-block="172.16.0.128/28" --full-management
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-7970/locations/northamerica-northeast1/operations/operation-1711086561504-6143962adaeaf-52b17a8a-23d9a574] to comple
te...working   
2249
...failed.                                                                                                                                               
ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1711086568132-614396312cf52-a6bfe5e9-1801c561]: errored while waiting for operation: projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1711086568132-614396312cf52-a6bfe5e9-1801c561: Operation failed with error: 
generic::invalid_argument: terraform apply failed, error: exit status 1, stderr: 

Error: Error creating Cluster: googleapi: Error 404: Not found: project "kcc-oi-7970" does not have a subnetwork named "kcc-sn" in region "northamerica-northeast1".
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.DebugInfo",
    "detail": "NOT_FOUND: not found: project \"kcc-oi-7970\" does not have a subnetwork named \"kcc-sn\" in region \"northamerica-northeast1\"",
    "stackEntries": [

checked vars.sh

NETWORK=kcc-ls-vpc
SUBNET=kcc-ls-sn

rerun

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ NETWORK=kcc-ls-vpc
SUBNET=kcc-ls-sn
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ echo "Creating Anthos KCC autopilot cluster ${CLUSTER} in region ${REGION} in subnet ${SUBNET} off VPC ${NETWORK}"
Creating Anthos KCC autopilot cluster kcc in region northamerica-northeast1 in subnet kcc-ls-sn off VPC kcc-ls-vpc
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET" --master-ipv4-cidr-block="172.16.0.128/28" --full-management
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-7970/locations/northamerica-northeast1/operations/operation-1711086838198-61439732bb1d4-230b07fc-dbefcb40] to comple
te...working...                   
2254

coming up

Screenshot 2024-03-21 at 22 54 52

33% 2258

the new org constraint forced on projects is causing an issue

Constraint constraints/compute.restrictVpcPeering violated for project 729005816584. Peering the network projects/gke-prod-na-ne1-dd32/global/networks/gke-n25d53e7a23908121151-bea7-3124-net is not allowed.

...failed.                                                                                                                                               
ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1711086845035-614397394034b-61a56dd6-6c5c0b5e]: errored while waiting for operation: projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1711086845035-614397394034b-61a56dd6-6c5c0b5e: Operation failed with error: 
generic::invalid_argument: terraform apply failed, error: exit status 1, stderr: 

Error: Error waiting for creating GKE cluster: Constraint constraints/compute.restrictVpcPeering violated for project 729005816584. Peering the network projects/gke-prod-na-ne1-dd32/global/networks/gke-n25d53e7a23908121151-bea7-3124-net is not allowed.

  on main_autopilot.tf line 32, in resource "google_container_cluster" "acp_cluster":
  32: resource "google_container_cluster" "acp_cluster" {

, stdout: 
google_container_cluster.acp_cluster: Creating...
google_container_cluster.acp_cluster: Still creating... [10s elapsed]
google_container_cluster.acp_cluster: Still creating... [20s elapsed]
google_container_cluster.acp_cluster: Still creating... [30s elapsed]
google_container_cluster.acp_cluster: Still creating... [40s elapsed]
google_container_cluster.acp_cluster: Still creating... [50s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [7m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [7m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [7m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [7m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [7m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [7m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [8m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [8m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [8m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [8m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [8m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [8m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [9m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [9m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [9m20s elapsed]

Subsequent cleanup succeeded
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$

Adding override

Screenshot 2024-03-21 at 23 11 46 Screenshot 2024-03-21 at 23 12 37

2313 rerun

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET" --master-ipv4-cidr-block="172.16.0.128/28" --full-management
Create request issued for: [kcc]
Waiting for operation [projects/kcc-oi-7970/locations/northamerica-northeast1/operations/operation-1711087974997-61439b6edddd2-ee8a3043-f8d2c51e] to comple
te...working

2319 67%

Screenshot 2024-03-21 at 23 18 40

2329 15 of 20 workloads

Screenshot 2024-03-21 at 23 30 38

check object browser - empty as expected

finish config controller config then kpt apply again

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ gcloud anthos config controller get-credentials "$CLUSTER" --location "$REGION"
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ gcloud anthos config controller get-credentials "$CLUSTER" --location "$REGION"
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$  gcloud anthos config controller list
NAME: kcc
LOCATION: northamerica-northeast1
STATE: RUNNING

reapply yamls

michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$  REL_SUB_PACKAGE="core-landing-zone"
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$ kpt live apply $REL_SUB_PACKAGE --reconcile-timeout=15m --output=table
Screenshot 2024-03-21 at 23 40 28

2340

Screenshot 2024-03-21 at 23 41 24

TRiage unmanaged resources - check existence first

Screenshot 2024-03-21 at 23 55 17 
michael@cloudshell:~/kcc-oi-20231206/kpt (kcc-oi-7970)$  kpt live status $REL_SUB_PACKAGE --inv-type remote --statuses InProgress,NotFound
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/security-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/org-log-sink-security-logging-project-oi0130 is not found
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-log-sink is not found
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/platform-and-component-services-infra-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/platform-and-component-services-infra-log-sink is not found
inventory-36746767/iampartialpolicy.iam.cnrm.cloud.google.com/projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions is InProgress: reference LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink is not found
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/logging-project-oi0130-data-access-sink is NotFound: Resource not found
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-log-sink is NotFound: Resource not found
inventory-36746767/dnsmanagedzone.dns.cnrm.cloud.google.com/networking/dns-project-oi0130-standard-core-public-dns is NotFound: Resource not found
inventory-36746767/service.serviceusage.cnrm.cloud.google.com/projects/dns-project-oi0130-dns is NotFound: Resource not found
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/platform-and-component-services-infra-log-sink is NotFound: Resource not found
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/mgmt-project-cluster-platform-and-component-log-sink is NotFound: Resource not found
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-security-logging-project-oi0130 is NotFound: Resource not found
inventory-36746767/logginglogsink.logging.cnrm.cloud.google.com/logging/org-log-sink-data-access-logging-project-oi0130 is NotFound: Resource not found
fmichaelobrien commented 3 months ago

Test out full scenario with 4 core packages (core-lz, client-lz, client-package.., client-setup https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#full-client-package-structure

I have one org up with 4 above + hub-env to test

in private get the list of objects via all the namespaces - example from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#deployment-verifying-kubernetes-gcp-resources

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get namespaces
NAME                              STATUS   AGE
...
hierarchy                         Active   67d
logging                           Active   67d
networking                        Active   67d
policies                          Active   67d
projects                          Active   67d

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n projects | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n hierarchy | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n logging | grep False