hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env

obriensystems commented 4 months ago

Likely due to the recent - couple of weeks ago - forced organization policy additions across GCP orgs - investigating/documenting (will need to check all deployed landing zones as this will only occur on redeployment or new GKE config connector object browser applies)

Part of landing zone automation in #766 and the older #446 see PR https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/846

Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env

First time redeploying the projects/hub-env package on top of the full 4 package core-lz, client*3

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet

org level

project level - example at the client-landing-zone host-project

need the same for hub-env

obriensystems commented 4 months ago

we need the same project level override on the org policy that cloud-landing-zone has for it's host project https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/client-folder/standard/applications-infrastructure/host-project/org-policies/exceptions/compute-restrict-cloud-nat-usage-except-host-project.yaml#L39

place it on https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env/org-policies/exceptions

with an additional

  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/hub-project-id # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}

obriensystems commented 4 months ago

Testing

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#########
#
# GCP Organization Policies
# Org policies that correspond with a Guardrail will contain a label indicating what Guardrails it helps in enforcing
# https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints
#
# Constraint: constraints/compute.restrictCloudNATUsage
#
# This list constraint defines the set of subnetworks that are allowed to use Cloud NAT.
#
# This exception is for the host project as it requires use of Cloud NAT.
#
#########
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${host-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/hub-project-id # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/PROJECT_ID # kpt-set: under:projects/${hub-project-id}}
  projectRef:
    external: "0000000000" # kpt-set: ${hub-project-id}}

obriensystems commented 4 months ago

rendering/applying change

michael@cloudshell:~/kcc-cso (kcc-cso-4380)$ ls
github  kpt
michael@cloudshell:~/kcc-cso (kcc-cso-4380)$ cd kpt/
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render hub-env --truncate-output=false

Successfully executed 2 function(s) in 1 package(s).
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply hub-env --reconcile-timeout=15m --output=table

missed one
  name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${host-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-cso2 # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
    cnrm.cloud.google.com/blueprint: 'kpt-fn'
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/xxdmu-admin1-hub-cso2} # kpt-set: under:projects/${hub-project-id}}
  projectRef:
    external: "xxdmu-admin1-hub-cso2}" # kpt-set: ${hub-project-id}}

fixed

 name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render hub-env --truncate-output=false
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply hub-env --reconcile-timeout=15m --output=table

checking

restarting vms

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest

found it - double end bracket
        - under:projects/xxdmu-admin1-hub-cso2} # kpt-set: under:projects/${hub-project-id}}

rerun
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt fn render hub-env --truncate-output=false
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live apply hub-env --reconcile-timeout=15m --output=table

never mind - found second typo

        - under:projects/xxdmu-admin1-hub-cso2 # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "xxdmu-admin1-hub-cso2}" # kpt-set: ${hub-project-id}}

fixed projectRef: - rerunning

failed

this is what we have
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-cso2 # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
    cnrm.cloud.google.com/blueprint: 'kpt-fn'
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/xxdmu-admin1-hub-cso2 # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "xxdmu-admin1-hub-cso2" # kpt-set: ${hub-project-id}

this is a working reference
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata: # kpt-merge: policies/compute-restrict-cloud-nat-usage-except-host-project-id
  name: compute-restrict-cloud-nat-usage-except-net-host-project-cso3 # kpt-set: compute-restrict-cloud-nat-usage-except-${host-project-id}
  namespace: policies
  labels:
    guardrail: "false"
  annotations:
    internal.kpt.dev/upstream-identifier: 'resourcemanager.cnrm.cloud.google.com|ResourceManagerPolicy|policies|compute-restrict-cloud-nat-usage-except-host-project-id'
    cnrm.cloud.google.com/blueprint: 'kpt-pkg-fn-live'
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/net-host-project-cso3 # kpt-set: under:projects/${host-project-id}
  projectRef:
    external: "net-host-project-cso3" # kpt-set: ${host-project-id}

metadata different
add kpt-merge: policies/compute-restrict-cloud-nat-usage-except-host-project-id
remove depends-on

obriensystems commented 4 months ago

removing kpt folder yaml - recopy/render to fix collision in policies namespace to fix

-  name: compute-restrict-cloud-nat-usage-except-host-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}
+  name: compute-restrict-cloud-nat-usage-except-hub-project-id # kpt-set: compute-restrict-cloud-nat-usage-except-${hub-project-id}

resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage                                                       21d   True    UpToDate       21d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-kcc-cso-4380                                   21d   True    UpToDate       21d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-net-host-project-cso3                          15d   True    UpToDate       15d
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2                          39m   False   UpdateFailed   39m

render
apply

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Failed: Update call failed: error applying desired state: summary: Error creating RouterNat: googleapi: Error 412: Constraint constraints/compute.restrictCloudNATUsage violated for projects/xxdmu-admin1-hub-cso2. projects/xxdmu-admin1-hub-cso2/regions/northamerica-northeast1/subnetworks/nane1-external-paz-snet is not allowed to use Cloud NAT., conditionNotMet
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest

checking gke object browser - still on old version - deleting manually

obriensystems commented 4 months ago

repasting clean yaml to kpt folder, render, apply (renamed file host to hub)

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except2-hub-project-id # kpt-set: compute-restrict-cloud-nat-usage-except2-${hub-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/hub-project-id # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/PROJECT_ID # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "0000000000" # kpt-set: ${hub-project-id}

rendered

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: ResourceManagerPolicy
metadata:
  name: compute-restrict-cloud-nat-usage-except2-xxdmu-admin1-hub-cso2 # kpt-set: compute-restrict-cloud-nat-usage-except2-${hub-project-id}
  namespace: policies
  annotations:
    config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-cso2 # kpt-set: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/${hub-project-id}
    cnrm.cloud.google.com/blueprint: 'kpt-fn'
  labels:
    guardrail: "false"
spec:
  constraint: "constraints/compute.restrictCloudNATUsage"
  listPolicy:
    allow:
      values:
        - under:projects/xxdmu-admin1-hub-cso2 # kpt-set: under:projects/${hub-project-id}
  projectRef:
    external: "xxdmu-admin1-hub-cso2" # kpt-set: ${hub-project-id}

fixed - but remove older gke object

ichael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep nat
inventory-89239324/computerouternat.compute.cnrm.cloud.google.com/networking/hub-nane1-external-nat is Current: Resource is Current
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except2-xxdmu-admin1-hub-cso2 is Current: Resource is Current

obriensystems commented 4 months ago

only the trailing original resource left - the nat is fixed

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt live status hub-env | grep Failed
inventory-89239324/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-restrict-cloud-nat-usage-except-xxdmu-admin1-hub-cso2 is Failed: Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Organization policy for projects/xxdmu-admin1-hub-cso2}: googleapi: Error 400: Request contains an invalid argument., badRequest

obriensystems commented 4 months ago

branch https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh837-natpolicyforhub PR https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/846

obriensystems commented 4 months ago

gcp

gke object

GoogleCloudPlatform / pubsec-declarative-toolkit

hub-env: Add a project level organization policy override for constraints/compute.restrictCloudNATUsage when deploying hub-env #837