Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.22k stars 592 forks source link

Gateway API httpRoute sectionName don't seem to do anything unless `hostname` is defined #6306

Open NissesSenap opened 3 months ago

NissesSenap commented 3 months ago

Is there an existing issue for this?

Current Behavior

Unless I specify hostname when having multiple listeners defined in kind Gateway, the routing of TLS don't seem to work, and instead I get the following error in the controller.

2024-07-09T13:49:34Z    error   Same SNI requested for multiple certs, can only serve one cert  {"served_secret_cert": "b8bde1a7-a695-4cd0-aa6f-50486bfefd91", "requested_secret_cert": "83d4728d-693c-4f7a-b013-069bc819909e", "sni": "*"}

Expected Behavior

Since I'm already defining sectionName in my httpRoute object, Kong shoulden't get any issues with the same SNI.

Example yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: podinfo
  namespace: staging
spec:
  hostnames:
  - podinfo.example.io
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong
    namespace: kong
    sectionName: https-io
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: podinfo
      port: 9898
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

Steps To Reproduce

Install KIC
Install Gateway API CRD
Create a GatewayClass
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: "true"
spec:
  controllerName: konghq.com/kic-gateway-controller

And a Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: example-com
            kind: Secret
            group: ""
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
        namespaces:
          from: All
    - name: https-io
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: example-io
            kind: Secret
            group: ""
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
        namespaces:
          from: All

Notice that hostname: "*.example.com" is missing from both the listeners This will give you the following error:

2024-07-09T13:49:34Z    error   Same SNI requested for multiple certs, can only serve one cert  {"served_secret_cert": "b8bde1a7-a695-4cd0-aa6f-50486bfefd91", "requested_secret_cert": "83d4728d-693c-4f7a-b013-069bc819909e", "sni": "*"}

This is even without having a httpRoute defined (which isn't strange from a code point of view). But if we look at the httpRoute object defined above we already use

  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: kong
    namespace: kong
    sectionName: https-io

So why would kong think that it's the same SNI requested for multiple certs?


### Kong Ingress Controller version

```shell
v3.2

Kubernetes version

➜ k version
Client Version: v1.29.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.5-gke.1091002

Anything else?

I can imagine this being a pain to solve, and a workaround could be to just document how this works. I'm wondering how Gateway API think this should work. I haven't looked at this closer, but here you can find the docs: https://gateway-api.sigs.k8s.io/guides/tls/?h=tls#targetrefs-and-tls

NissesSenap commented 3 months ago

For anyone finding this issue, there is a workaround for this issue. Which is that you have to define the hostname in your gateway.

For example

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: example-com
            kind: Secret
            group: ""
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
        namespaces:
          from: All
    - name: https-io
      protocol: HTTPS
      hostname: "*.example.io"
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: example-io
            kind: Secret
            group: ""
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
        namespaces:
          from: All
NissesSenap commented 3 months ago

I read through the Gateway API documentation a few more times, and my understanding now is that unless you define a hostname in the Gateway, it is seen as the default TLS cert. So I can understand why Kong acts this way, when I had two listeners with no hostname defined.

Obviously, I think this is a bit error-prone, and it would be good to document this very clearly in Kong as well. The only way why I figured it out was from reading the gateway-api docs.

I might create an issue in Gateway API about this, since I personally think it shouldn't be possible to do the error I just did. I think it will be hard to write a good CEL rule to make sure this doesn't happen, but who knows. Another option could be to clarify this a bit within Gateway API, to make sure this is well-defined how the controllers should work.

I will leave the issue open, since I think it's a good thing to document.