ssc-spc-ccoe-cei / gcp-tools

This repo is meant to be a submodule of the gcp-repo-template repository. https://github.com/ssc-spc-ccoe-cei/gcp-repo-template
3 stars 3 forks source link

New Organization Day 0: walkthrough/automation example KCC LZ install using the setup-kcc.sh script with sourced env.sh readme instructions #32

Open fmichaelobrien opened 1 year ago

fmichaelobrien commented 1 year ago

assigned to fmichaelobrien https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh

Issues pending

admin_@cloudshell:~/pdt-arg$ mkdir ssc-spc-ccoe-cei
admin_@cloudshell:~/pdt-arg$ cd ssc-spc-ccoe-cei/
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei$ git clone https://github.com/CloudLandingZone/gcp-tools.git
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei$ cd gcp-tools/scripts/bootstrap/
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ cp .env.sample pdt-arg.env

admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ export PROJECT_ID=pdt-arg
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ export BILLING_ID=$(gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' | sed 's/.*\///')
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ echo $BILLING_ID
01A5

admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ chmod 755 setup-kcc.sh 
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ ./setup-kcc.sh pdt-arg.env 

edit pdt-arg.env

export CLUSTER=pdt-arg
export REGION=northamerica-northeast1
export PROJECT_ID=pdt-arg-kcc1
export LZ_FOLDER_NAME=pdt-arg-kcc
export NETWORK=pdt-arg
export SUBNET=pdt-arg-sn
export ORG_ID=226082700214
export ROOT_FOLDER_ID= # This one is only required if not deploying at the org level. Ex. for testing. See option 2 when executing the Config Controller project and cluster below
#export BILLING_ID=<Billing ID>
export GIT_USERNAME=obriensystems #<Git-Username> # For Azure Devops, this is the name of the Organization
export CONFIG_SYNC_REPO=pdt-arg-kcc # <Repo for Config Sync> # should default to tier1 config sync
export CONFIG_SYNC_VERSION='HEAD'
export CONFIG_SYNC_DIR=deploy/dev #<Directory for config sync repo which syncs> # Should default to deploy/<env>

run

admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg)$ ./setup-kcc.sh pdt-arg.env 
Waiting for [operations/cf.6355467925699962575] to finish...done.                                                                                                                                    
Created [<Folder
 createTime: '2023-05-23T13:43:04.314Z'
 displayName: 'pdt-arg-kcc'
 lifecycleState: LifecycleStateValueValuesEnum(ACTIVE, 1)
 name: 'folders/334575950523'
 parent: 'organizations/226082700214'>].
folders/334575950523
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/pdt-arg-kcc1].
Waiting for [operations/cp.8136442715168594598] to finish...done.              

Updated property [core/project].
ERROR: (gcloud.services.enable) The operation "operations/acf.p2-769094436748-cc561e42-d01c-42a8-b504-b30205f56ceb" resulted in a failure "[service container.googleapis.com encountered internal erro

comment to line 30 put a sleep 20 sec

gcloud services enable krmapihosting.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com cloudbilling.googleapis.com serviceusage.googleapis.com servicedirectory.googleapis.com dns.googleapis.com

admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg-kcc1)$ ./setup-kcc.sh pdt-arg.env 
Operation "operations/acf.p2-769094436748-a1cc938d-4f2c-4305-aea1-ca4ba8ba77e8" finished successfully.
Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc1/global/networks/pdt-arg].
NAME: pdt-arg
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt-arg --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt-arg --allow tcp:22,tcp:3389,icmp

Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc1/regions/northamerica-northeast1/subnetworks/pdt-arg-sn].
NAME: pdt-arg-sn
REGION: northamerica-northeast1
NETWORK: pdt-arg
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
Creating router [kcc-router]...done.  

Create request issued for: [pdt-arg]
Waiting for operation [projects/pdt-arg-kcc1/locations/northamerica-northeast1/operations/operation-1684849746948-5fc5ca6aab312-73e55bb9-0f150025] to complete...working       

check cluste

Create request issued for: [pdt-arg]
Waiting for operation [projects/pdt-arg-kcc1/locations/northamerica-northeast1/operations/operation-1684849746948-5fc5ca6aab312-73e55bb9-0f150025] to complete...failed.                             
ERROR: (gcloud.anthos.config.controller.create) unexpected error occurred while waiting for SLM operation [projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1684849754561-5fc5ca71eddb7-e8a08e34-6f7943fe]: errored while waiting for operation: projects/krmapihosting-slm/locations/northamerica-northeast1/operations/operation-1684849754561-5fc5ca71eddb7-e8a08e34-6f7943fe: Operation failed with error: 
generic::invalid_argument: terraform apply failed, error: exit status 1, stderr: 
2023/05/23 06:49:38 [DEBUG] Using modified User-Agent: Terraform/0.12.31 Cloud SSA

Error: Error waiting for creating GKE cluster: 
        (1) Not all instances running in IGM after 19.547914969s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-arg-default-pool-b39e8300-n97n' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/pdt-arg-kcc1. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information
        (2) Not all instances running in IGM after 21.667824619s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-arg-default-pool-6a80d768-f0nk' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/pdt-arg-kcc1. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information
        (3) Not all instances running in IGM after 23.821246077s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-arg-default-pool-0a699d53-bqwt' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/pdt-arg-kcc1. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.

  on main_gke.tf line 32, in resource "google_container_cluster" "acp_cluster":
  32: resource "google_container_cluster" "acp_cluster" {

, stdout: 
google_container_cluster.acp_cluster: Creating...
google_container_cluster.acp_cluster: Still creating... [10s elapsed]
google_container_cluster.acp_cluster: Still creating... [20s elapsed]
google_container_cluster.acp_cluster: Still creating... [30s elapsed]
google_container_cluster.acp_cluster: Still creating... [40s elapsed]
google_container_cluster.acp_cluster: Still creating... [50s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [1m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [2m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [3m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [4m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [5m50s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m0s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m10s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m20s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m30s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m40s elapsed]
google_container_cluster.acp_cluster: Still creating... [6m50s elapsed]

Warning: Interpolation-only expressions are deprecated

  on main_gke.tf line 143, in resource "google_container_node_pool" "acp_pool":
 143:     machine_type = "${var.node_machine_type}"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a
mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.

Subsequent cleanup succeeded
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg-kcc1)$                                                                                                                                                                                      

check cluster and continue script after line 105

secure boot is not enabled
       (1) Not all instances running in IGM after 19.547914969s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-arg-default-pool-b39e8300-n97n' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/pdt-arg-kcc1. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information

gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET"

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/344#issuecomment-1600993940

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/132

rerun script past kcc cluster after setting secure boot policy override - constraints/compute.requireShieldedVm

Screenshot 2023-05-23 at 10 13 41 AM
admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg-kcc1)$ ./setup-kcc.sh pdt-arg.env 
Create request issued for: [pdt-arg]
Waiting for operation [projects/pdt-arg-kcc1/locations/northamerica-northeast1/operations/operation-1684851346311-5fc5d05ff0772-9eb3ed39-b076fbb7] to complete...working..                           

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/132 KCC cluster up

Screenshot 2023-05-23 at 10 35 09 AM

continue script

gcloud anthos config controller get-credentials "$CLUSTER" --location "$REGION"

dmin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg-kcc1)$ ./setup-kcc.sh pdt-arg.env 
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-pdt-arg.
Updated IAM policy for organization [226082700214].

storageLocation: northamerica-northeast1
secret/git-creds created
rootsync.configsync.gke.io/root-sync created

##WARNING - The root-sync.yaml file should be checked into the <tier1_infra-REPO>
fmichaelobrien commented 1 year ago

rerun

admin_@cloudshell:~/pdt-arg/ssc-spc-ccoe-cei/gcp-tools/scripts/bootstrap (pdt-arg-kcc10)$ ./setup-kcc.sh pdt-arg.env 
Waiting for [operations/cf.9135754359470030039] to finish...done.                                                                                                                                    
Created [<Folder
 createTime: '2023-06-21T15:39:42.090Z'
 displayName: 'pdt-arg-kcc11'
 lifecycleState: LifecycleStateValueValuesEnum(ACTIVE, 1)
 name: 'folders/872374816049'
 parent: 'organizations/226082700214'>].
folders/872374816049
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/pdt-arg-kcc11].
Waiting for [operations/cp.6096008890562947989] to finish...done.                                                                                                                                    
Enabling service [cloudapis.googleapis.com] on project [pdt-arg-kcc11]...
Operation "operations/acat.p2-831930126559-7d3397aa-1f57-42d1-94c5-d4f202782516" finished successfully.
Updated property [core/project] to [pdt-arg-kcc11].
billingAccountName: billingAccounts/01....82
billingEnabled: true
name: projects/pdt-arg-kcc11/billingInfo
projectId: pdt-arg-kcc11
Updated property [core/project].
Operation "operations/acf.p2-831930126559-170db725-4b96-4d2a-83ec-71579108e353" finished successfully.
Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/networks/pdt-arg].
NAME: pdt-arg
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE: 
GATEWAY_IPV4: 

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt-arg --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt-arg --allow tcp:22,tcp:3389,icmp

Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/regions/northamerica-northeast1/subnetworks/pdt-arg-sn].
NAME: pdt-arg-sn
REGION: northamerica-northeast1
NETWORK: pdt-arg
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE: 
INTERNAL_IPV6_PREFIX: 
EXTERNAL_IPV6_PREFIX: 
Creating router [kcc-router]...done.                                                                                                                                                                 
NAME: kcc-router
REGION: northamerica-northeast1
NETWORK: pdt-arg
Creating NAT [kcc-router] in router [kcc-router]...done.                                                                                                                                             
Created Policy [https://dns.googleapis.com/dns/v1/projects/pdt-arg-kcc11/policies/dnspolicy1].
{
  "description": "dns policy to enable logging",
  "enableInboundForwarding": false,
  "enableLogging": true,
  "id": "1134094650152878946",
  "kind": "dns#policy",
  "name": "dnspolicy1",
  "networks": [
    {
      "kind": "dns#policyNetwork",
      "networkUrl": "https://compute.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/networks/pdt-arg"
    }
  ]
}
Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/addresses/apis-private-ip].
Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/forwardingRules/endpoint1].
Created [https://dns.googleapis.com/dns/v1/projects/pdt-arg-kcc11/managedZones/googleapis].
NAME: googleapis.com.
TYPE: A
TTL: 300
DATA: 10.255.255.254
NAME: *.googleapis.com.
TYPE: CNAME
TTL: 300
DATA: googleapis.com.
Created [https://dns.googleapis.com/dns/v1/projects/pdt-arg-kcc11/managedZones/gcrio].
NAME: gcr.io.
TYPE: A
TTL: 300
DATA: 10.255.255.254
NAME: *.gcr.io.
TYPE: CNAME
TTL: 300
DATA: gcr.io.
Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/firewalls/allow-egress-azure].                                                             
Creating firewall...done.                                                                                                                                                                            
NAME: allow-egress-azure
NETWORK: pdt-arg
DIRECTION: EGRESS
PRIORITY: 5000
ALLOW: tcp:22,tcp:443
DENY: 
DISABLED: False
Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/firewalls/allow-egress-github].                                                            
Creating firewall...done.                                                                                                                                                                            
NAME: allow-egress-github
NETWORK: pdt-arg
DIRECTION: EGRESS
PRIORITY: 5001
ALLOW: tcp:22,tcp:443
DENY: 
DISABLED: False
Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/firewalls/allow-egress-internal].                                                          
Creating firewall...done.                                                                                                                                                                            
NAME: allow-egress-internal
NETWORK: pdt-arg
DIRECTION: EGRESS
PRIORITY: 1000
ALLOW: all
DENY: 
DISABLED: False
Creating firewall...working..Created [https://www.googleapis.com/compute/v1/projects/pdt-arg-kcc11/global/firewalls/deny-egress-internet].                                                           
Creating firewall...done.                                                                                                                                                                            
NAME: deny-egress-internet
NETWORK: pdt-arg
DIRECTION: EGRESS
PRIORITY: 65535
ALLOW: 
DENY: all
DISABLED: False
Create request issued for: [pdt-arg]
Waiting for operation [projects/pdt-arg-kcc11/locations/northamerica-northeast1/operations/operation-1687362167709-5fea59e9d1eb9-76df9161-d9eb9ad4] to complete...working.                           
Screenshot 2023-06-21 at 12 00 12 PM
fmichaelobrien commented 1 year ago

moving back to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/296 and installing min LZ

obriensystems commented 9 months ago

20231019 setup.sh script for kcc cluster delete/recreate and lz kpt apply/destroy in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/446#issuecomment-1771365186 under https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh446-hub/solutions/setup.sh

obriensystems commented 9 months ago

Actually for 1 of the 2 - the historical Shielded - we are good with the following override

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp -n policies
NAME                                                                                                         AGE     READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project   9m36s   True    UpToDate   9m8s