Istio Gateway resource should be discoverable using a label

[fourthisle]

Description:

Currently, the watcher reconciler expects that an istio gateway resource with the name lifecycle-manager-gateway is deployed in the default namespace. Due to the fact that this can vary between the different KCP environments, a better approach would be to implement a more generic way for resolving the name and namespace of the gateway resource. The watcher reconciler uses it to configure the virtual service resource it is managing.

Edit: In this Issue we don't configure the gateway itself. It's only about finding a proper gateway and configuring Virtual Service for the Watcher using that Gateway.

ACs:

• [x] Design more flexible Gateway lookup process: for example a label selector to resolve the istio gateway resource used by the custom istio client.
• [x] Make the Gateway configuration more dynamic - command line flags require application restart to change. Maybe the Gateway configuration can be fetched from the Watcher object?
• [ ] Use operator.kyma-project.io/ingress-for + controller name on the gateway to determine the controller that should be used
• [x] If there are more than 1 candidate, emit a warning-event for the watcher object and default to the first found gateway
• [x] If there is no gateway candidate, return an error (stop processing) and output a warning-event for the watcher object that there is no ingress for events

Related:

_Originally posted by @jakobmoellersap in https://github.com/kyma-project/lifecycle-manager/pull/231#discussion_r987619930_

[Tomasz-Smelcerz-SAP]

@khlifi411 I have a question about the AC: "Use operator.kyma-project.io/ingress-for + controller name on the gateway to determine the controller that should be used" I assume this means, that spec.selector.app should be set to the operator.kyma-project.io/ingress-for-<controller-name> on the gateway object. But we don't create any gateway object directly in the codebase. I assume then that this should be done in the yaml files / kustomization definitions of the project. Is that correct?

[jeremyharisch]

@Tomasz-Smelcerz-SAP the Gateway will be created at Deploy-Time. I am currently working on that. You do not have to implement the Gateway for this task 👍

[fourthisle]

@khlifi411 I have a question about the AC: "Use operator.kyma-project.io/ingress-for + controller name on the gateway to determine the controller that should be used" I assume this means, that spec.selector.app should be set to the operator.kyma-project.io/ingress-for-<controller-name> on the gateway object. But we don't create any gateway object directly in the codebase. I assume then that this should be done in the yaml files / kustomization definitions of the project. Is that correct?

Yes. That's right!

[Tomasz-Smelcerz-SAP]

@khlifi411 @jeremyharisch One more question. What kind of label selector are we targeting? I need a default. Should it be just the label (we don't care for value), like operator.kyma-project.io/gateway, or maybe key=value, like operator.kyma-project.io/gateway=default ? Any hints?

[fourthisle]

@khlifi411 @jeremyharisch One more question. What kind of label selector are we targeting? I need a default. Should it be just the label (we don't care for value), like operator.kyma-project.io/gateway, or maybe key=value, like operator.kyma-project.io/gateway=default ? Any hints?

For me, It is better to check for just a label as checking for the value assumes that other gateways deployed on KCP have different values for the label. My suggestion would be operator.kyma-project.io/default-gateway 👍

[jeremyharisch]

@khlifi411 @jeremyharisch One more question. What kind of label selector are we targeting? I need a default. Should it be just the label (we don't care for value), like operator.kyma-project.io/gateway, or maybe key=value, like operator.kyma-project.io/gateway=default ? Any hints?

@khlifi411 @Tomasz-Smelcerz-SAP I would like that we stick to Kubernetes "standards" meaning looking for a label with a specific value. Since this does not hurt us, and we do not start introducing random label values. But in general, either way works for our setup.