search

GoogleCloudPlatform / k8s-multicluster-ingress

kubemci: Command line tool to configure L7 load balancers using multiple kubernetes clusters
Apache License 2.0
377 stars 68 forks source link

Error Ensuring Firewall rules when running on Clusters in a Shared VPC #217

Open boredabdel opened 5 years ago

boredabdel commented 5 years ago

Hi,

I'm having an issue with kubemci trying to create a multi cluster ingress across two clusters that are attached to a sharedvpc.

Shared VPC in project A. This is the Host project Cluster 1 in project B Attached to the Shared VPC in project A Cluster 2 in project B Attached to the Shared VPC in project A

Same service running on both clusters listening to the same node port.

I0227 15:24:45.316852 52427 gen.go:5430] GCEFirewalls.Insert(context.Background, Key{"mci1-fr--nginx-mci"}, ...): projectID = PROJECT_B, rk = &{ProjectID:PROJECT_B Operation:Insert Version:ga Service:Firewalls} I0227 15:24:45.408566 52427 gen.go:5441] GCEFirewalls.Insert(context.Background, Key{"mci1-fr--nginx-mci"}, ...) = googleapi: Error 400: Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1/projects/PROJECT_A/global/networks/PROJECT_A'. Cross project referencing is not allowed for this resource., invalid Error ensuring firewall rule: Error googleapi: Error 400: Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1/projects/PROJECT_A/global/networks/PROJECT_A . Cross project referencing is not allowed for this resource., invalid in ensuring firewall rule Error: Error in creating load balancer: 1 error occurred:

What i think happens is the following

1 - kubemci check the project hosting the cluster (PROJECT_B in my case) for a specific firewall rule. 2 - The call fails, so it calls the EnsureFirewallRule function to create those. 3 - The call fails because the API called behind apparently doesn't support creating a firewall rule in a different project then the one where the call is made. hence the error "Cross project referencing is not allowed for this resource"

I understand that kubemci is trying to create fw rules for the LB Health Check and Backend Service IP Ranges. But we could create those in the shared VPC project manually or via a script. will it be possible to have a flag to kubemci to skip trying to create firewall rules ?

jawlitkp commented 5 years ago

Having the same issue. please provide work around

boredabdel commented 5 years ago

This have been fixed already.

You just need to add the following annotation to the ingress definition to silence the alerts about firewall rules.

networking.gke.io/suppress-firewall-xpn-error: "true"

boredabdel commented 5 years ago

Got confused with this and an other issue, reopening as we do need a workaround for this issue,

artazar commented 4 years ago

I am facing exactly the same issue, is there any known workaround available?

nikhiljindal commented 4 years ago

Yes this is a known issue. kubemci does not support XPN: https://github.com/GoogleCloudPlatform/k8s-multicluster-ingress/issues/56.

I havent tried them myself, but 2 potential workarounds I can think of are:

  1. Create the Firewall rule manually. Or
  2. Try the patch: https://github.com/GoogleCloudPlatform/k8s-multicluster-ingress/pull/222

To create the firewall rule manually: Run the kubemci create command so that it creates all other resources. Then create the necessary firewall rule yourself. Ensure that you use the right project. You can create the firewall rule manually by using cloud console, or by running the appropriate gcloud command: https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/create.