GoogleCloudPlatform / pubsec-declarative-toolkit

The GCP PubSec Declarative Toolkit is a collection of declarative solutions to help you on your Journey to Google Cloud. Solutions are designed using Config Connector and deployed using Config Controller.

Apache License 2.0

30 stars 27 forks source link

restrictVpcPeering organization policy prevents the creation of private GKE/Autopilot Clusters #812

Closed KingBain closed 5 months ago

KingBain commented 5 months ago

As mentioned in #588 a policy blocks production teams from deploying private clusters.

This issue can be created with the command:

gcloud beta container clusters create-auto "autopilot-privatecluster-fail" \
--project "phx-gkeweirdpeer" \
--region "northamerica-northeast1" \
--network "projects/phx-gkeweirdpeer/global/networks/test" \
--subnetwork "projects/phx-gkeweirdpeer/regions/northamerica-northeast1/subnetworks/first" \
--enable-master-authorized-networks \
--enable-private-nodes \
--enable-private-endpoint

From the documentation, this is caused by how Google connects the shared control plane to the private cluster. Google manages the shared control plane in a GCP owned project.

Error can be seen in the screenshot as well as the Google owned network that is attempting to be peered.

Peering the network projects/gke-prod-na-ne1-2fbb/global/networks/gke-n965f1051bcd2d9c10f9-a2fa-7721-net is not allowed.

KingBain commented 5 months ago

This blog post speaks to a workaround where we need to whitelist Google owned projects that are used for private cluster peering https://autonomousthingz-life.medium.com/google-cloud-organisation-policies-get-the-restrict-vpc-peering-usage-organisation-policy-d87dce84a5e7

listPolicy:
  inheritFromParent: false
  allowedValues:
  # Required if projects use GKE private clusters.
  - under:folders/391150242170  # google.com/tenancy-units/gke/prod folderfor GKE master peering.

This doesn't seem like a great solution and I haven't tested it.

@obriensystems does this workaround still work ?

davelanglois-ssc commented 5 months ago

@KingBain I recommend defining the value below in your setters.yaml of the core-landing-zone package. Reach out to @fmichaelobrien or @tackaberry to obtain the value for yyyyyyyyy allowed-vpc-peering: |

"under:organizations/xxxxxxxxxxx" # your own GCP organization
"under:organizations/yyyyyyyyyy" # Google ORG where the projects hosting the GKE control plane reside. It also allows other Google Managed Services like Cloud IDS.

tackaberry commented 5 months ago

@KingBain @davelanglois-ssc I will send the Google Production Org in an email. I realize it isn't secret, I just avoid it being captured here.

obriensystems commented 3 months ago

Running into this on GKE anthos cluster recreation - not just workload GKE clusters - putting in a project override for now

794

hagarwalppro commented 3 months ago

Any help here ? We're also facing a similar issue while creation a composer instance with image composer-2.6.5-airflow-2.7.3 with autopilot cluster in a shared vpc at org level

Constraint constraints/compute.restrictVpcPeering violated for project xxxxxxxxxxx. Peering the network projects/gke-prod-europe-west3-ca61/global/networks/gke-n2be24655187ebf51da7-90f6-8f6a-net is not allowed

whitelisted the google folder also but doesn't help

@davelanglois-ssc @tackaberry I see you have asked to whitelist the whole Google org id here, can we get the org id please if that's the recommended way forward ?

KingBain commented 3 months ago

IMO this policy creates more problems than it fixes, we were able to get our issue resolved with the whitelisted policy method, but the solution feels like a kludge.

hagarwalppro commented 3 months ago

@KingBain can you please share more details on how you achieved it ?

KingBain commented 3 months ago

I'm waiting to see if @tackaberry will respond, he is the googler here.