"gcloud compute networks peerings update" command equivalent terraform resource cannot find (Cloud build private pools)

[sreenivas-ps]

Hi Team I'm working to automate the GCP's cloud build private pools using the instructions from here but I'm unable to find the terraform resource equivalent gcloud compute networks peerings update command which is needed to update the peering between the vpc network where gke cluster is and google side control plane vpc.

any help on this would be much appreciated.

Regards, Sreenivas

[edwardmedia]

@sreenivas-ps how did you create the peering? In terraform, there is google_compute_network_peering that can manage it and it does support export_custom_routes

[sreenivas-ps]

@edwardmedia : When you create a GKE cluster in google cloud platform, it automatically peers with google side vpc. Now I'd like to get that peering name from the cluster name and then update it. (the below commands are from the above link)

// Retrieve the name of the GKE cluster’s VPC Network Peering. This VPC Network Peering was automatically created when you created the GKE cluster.

export GKE_PEERING_NAME=$(gcloud container clusters describe PRIVATE_CLUSTER_NAME
  --region=REGION--format='value(privateClusterConfig.peeringName)')

// Enable the export of custom routes in order to advertise the private pool network to the GKE cluster control plane:

gcloud compute networks peerings update $GKE_PEERING_NAME
--network=GKE_VPC_NAME
--export-custom-routes
--no-export-subnet-routes-with-public-ip

Basically I created the gke cluster with terraform and now I need to update google side vpc peering name to export_custom_routes. I'm creating all the resources for cloud build private pools with terraform including the ha-vpn (ha-vpn module used from community provided one) but was stuck with gcloud compute networks peering update which I cannot find the terraform equivalent resource.

[edwardmedia]

@sreenivas-ps peering_name is available in Terraform after you create a private cluster. You should be able to retrieve it with output.

There is no update command from Terraform perspective. But if you put all in a pipeline, you may create a google_compute_network_peering first in an apply. Once the cluster is created, you may update google_compute_network_peering with another apply.

[sreenivas-ps]

@edwardmedia : I'm not creating any peering myself but the peering is automatically created by GKE service and services vpc-peerings itself.

I use this command: gcloud services vpc-peerings connect connects a private service connection to a service (servicenetworking.googleapis.com) via a VPC network which also creates peering automatically.

I need to run gcloud compute networks peerings update twice i.e. once on the GKE cluster control plane peered_network. Another time on peered_network that it created when I ran vpc-peerings connect command.

would it be possible we can have a quick meeting or a hangout.

[edwardmedia]

@sreenivas-ps I think this might not be a good use case for Terraform. Terraform works on comparing your state against the config. If there is a diff, terraform apply will bring the state to match your config. Do you see if that fits your case? As long as you are able to update the config accordingly in each step ( like in a pipeline), Terraform could work like update. The resource is there. Does this make sense?

[sreenivas-ps]

@edwardmedia : this is definitely a good use case for Terraform as setting up Cloud Builds Private Pools service is very complex. See the link in the post above. This service is something new in GCP and works fine with the gcloud compute networks peerings update commands so there should be an API support from google already. Hence this post if there is or would be support from terraform google provider, or anything planned on this?

[edwardmedia]

@slevenick What do you think?

[slevenick]

Have you tried importing the automatically created peerings using the google_compute_network_peering resource & then managing the peering via that resource? That would be the normal Terraform flow for managing an existing resource

It may be tricky if the GKE cluster keeps trying to manage the peering it created, but I'm not familiar with this interaction

[sreenivas-ps]

All resources are created using terraform and this is not about import an existing resources. I raised a support request with google and this is what they recommended:

the specialist team the recommendation is:

Not use terraform for this purpose, rather than getting whitelisted for beta API, you should use cloud console or gcloud to create this using GA API beta API , it is not recommended to be used since, it may have bugs and would not be fixed. And it might give you more issues than a fix.

Unfortunately we cannot offer anything better at this point until TF supports GA API, not estimated time when this would be available.

so for now I am using terraform null_resource to run the gcloud commands as part of creating all resources with terraform. I don't like to use null_resource. Also google needs to whitelist the gcp project to create workpools within that project which is again using a terraform null_resources.

[slevenick]

Sorry, I guess I don't understand what exactly is going on here.

So you are creating the peerings you are trying to update from within Terraform? This comment made me think they were not created through Terraform, and were instead created automatically when another resource was created

[bahag-hammera]

We ran into the same problem as described. We need something to update the "automatically created" network peering between our projects and the google provided project (used for the gke master node/controlplane). In our case, we try to deploy a provate gke cluster with routing over VPN. It's all working except the "export routes command" via terraform.

Right now we use a local exec like that: gcloud compute networks peerings update PEERING NAME

But the peering between our host project and the google provied project ist named like: gke-ndf56d7f019d79e601a0-13b6-6b03-peer -> So its quite impossible to get the correct name, to update/import it via https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network_peering_routes_config

Maybe thats make something a little bit clearer

[sreenivas-ps]

@bahag-hammera : you can get the peering_name from the terraform's output of the google_container_cluster

cluster_peering_name = (var.enable_private_nodes && length(google_container_cluster.primary.private_cluster_config) > 0) ? google_container_cluster.primary.private_cluster_config.0.peering_name : null

i'm using community provided cluster: "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"

did you get the whitelisting of your gcp project added by google? At least this wasn't whitelisted for me to create worker pool with terraform resource so i had to use null_resource for worker pool creation.

[sreenivas-ps]

@slevenick : sorry you misunderstood me from my comment above. All resources were created with terraform.