Remove tight coupling to ISTIO and api-gateway module (e.g. switch to `Warning` state when Istio module is missing)

[tobiscr]

Description

The application connector is currently reusing the certificate from the default gateway which leads to a tight coupling between the application connector and the api-gateway module.

As we have no mandatory modules in Kyma anymore, such coupling is not allowed and can lead to operational incidents if customers are not using the ISTIO / api-gateway modules from Kyma.

This is a high risk in regards to our operational robustness and we have to remove this dependency asap.

See example manifest from application-connector gateway and the used certificate secret:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  creationTimestamp: "2024-02-09T13:54:20Z"
  finalizers:
  - application-connector-manager.kyma-project.io/deletion-hook
  generation: 1
  labels:
    app.kubernetes.io/instance: application-connector
    app.kubernetes.io/managed-by: application-connector-manager
    app.kubernetes.io/name: application-connector
    app.kubernetes.io/part-of: application-connector-manager
    release: application-connector
  name: kyma-gateway-application-connector
  namespace: kyma-system
  resourceVersion: "248000886"
  uid: ...
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - gateway.....kyma.ondemand.com
    port:
      name: https-app-connector
      number: 443
      protocol: HTTPS
    tls:
      credentialName: kyma-gateway-certs
      mode: MUTUAL

We have to agree on a different mechanism and avoid to use the default-gateway certificate in the application-connector module:

• We will use our own logic to generate a gateway certificate (e.g. application-connector-gateway-certificate) and no longer use the one from the api-gateway

Options for solving this requirement is:

• Use Gardener cert-manager for creating our own certificate (this option needs to be verified and feasibility confirmed)

AC:

• [ ] The ACM is not longer using the gateway certfiicate form API-gateway module and creating its own gateway-certificate instance
• [ ] Ensure smooth migration is supported and existing application will not be disconnected before this change is rolled out
  • [ ] If downtime is not avoidable, schedule the rollout of this change for the next SKR maintenance window

Reason No dependencies to other modules as it's colliding with the technical agreements we made for Kyma modules.

[koala7659]

This is complicated topic, and we intentionally decided to go such way to reuse such kyma-gateway certificate in application-connector Istio gateway in order to enable MTL. And this approach was agreed by Goats and PB as far as I remember.

If they want us to do stop using that that might have a lot of consequences, and would require us to migrate ALL Kyma clusters to new certificates, and ALL connected application might also be affected. It is a big risk and I don't want to start working on this task without some research.

Possible solutions:

1) Workaround: would be now to not allow application connection module to run without api-gateway module enabled.

2) Remove coupling from both modules: Extract this gateway certificate from api-gateway module and put it into Kyma separately.

3) Separate certificate: Issue new certificate for application connector, and migrate all Kyma workloads (seems crazy task to me)

[tobiscr]

@pbochynski : any objections to establish the API-Gateway module as mandatory pre-requisite for the Application-Connector module?

Cleaner approach is to use a dedicated certificate, but the effort for the migration is quite high and we have to verify how it collaborates with the Compass Directory. This will require additional analysis.

[tobiscr]

A diagram is required showing the different scenarios:

• Old cluster without ISTIO
• Old cluster with ISTIO but without API-gateway
• Old cluster with ISTIO and with API-gateway
• New cluster without ISTIO
• New cluster with ISTIO but without API-gateway
• New cluster with ISTIO but with API-gateway

[tobiscr]

[tobiscr]

Fix will require to modularise the Compass Runtime Agent (move to ACM). Currently, the issue is not solvable in a meaningful way.

This incident is blocked by #114

[tobiscr]

See also #90 - could be related to this change!

[tobiscr]

We agreed to split this task in two phases:

Phase 1:

• Fix the issue which causes the module to switch into an Error state when Istio is missing. It has to switch into the state Warning and fire an expressive error-message which clearly indicates that Istio is missing

Phase 2:

• Verify how we can get rid of the tight coupling with the Kyma Istio module:
  • We should not rely that customers are using our Istio module. It has also to be possible to use "plain" Istio.
  • We have to get rid of any references which exist only when the Kyma Istio module is used. We have to focus to use "neutral" resources which can be created by ourself when Istio itself exists.

[tobiscr]

Identified solution COULD be related to https://github.com/kyma-project/compass-manager/issues/188 (has to be checked if Compass Runtime Agent is really using API-gateway module, respectively the kyma-gateway).