search

GoogleCloudPlatform / kubeflow-distribution

Blueprints for Deploying Kubeflow on Google Cloud Platform and Anthos
Apache License 2.0
80 stars 63 forks source link

Avoid duplicating the ISTIO Ingressgateway service when using ASM #22

Open jlewi opened 4 years ago

jlewi commented 4 years ago

When we hydrate the manifests for istio from the ISTIOControl Plane operator https://github.com/kubeflow/manifests/blob/master/gcp/v2/asm/istio-operator.yaml

We end up generating the following IngressGateway.yaml file IngressGateway.yaml.txt

This defines the service "istio-ingressgateway".

The problem is this K8s service doesn't contain the annotation 

beta.cloud.google.com/backend-config: '{"ports": {"http2":"iap-backendconfig"}}'

Which is needed to associate it with a backendconfig to configure IAP.

In the past we just duplicated this resource: https://github.com/kubeflow/manifests/blob/master/istio/iap-gateway/base/istio-ingressgateway.yaml

In general this worked because this would be applied after applying the ISTIO config.

With ACM this starts to be more problematic because we end up with two resources with the same name and ACM doesn't allow this.

There's a couple possible options.

  1. We could create a second ingressgateway K8s service with a different name and use that for our load balancer
  2. We could use a kustomize function to transform the existing ISTIO service and add the appropriate annotation

1 has the draw back that we risk getting out of sync with the configs generated by ISTIO ingressgateway.

2 Is better in this regard because it is more of a template free solution.

2 has the disadvantage though its depending on newer functionality in kustomize. I'm not even sure its available in of the existing releases (it is available on master).

Related to: #4

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
platform/gcp 0.94
kind/bug 0.64

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

jlewi commented 4 years ago

We should go with the kpt transform. We have started using kpt functions with ACM in order to remove the namespace.

A quick hack would be to use yq to change the gateway value; but its probably just a bit more work to create a kpt transform to change the gateway.

jlewi commented 4 years ago

See: kubeflow/manifests#1169 for information about which gateway we should be using.

jtfogarty commented 4 years ago

/priority p2