TLS configuration is invalid with custom TCP ports

[p1-repo-sync-bot[bot]]

Overview

When adding custom TCP ports to a gateway definition (as documented here), helm upgrade fails with the following error:

✗ Helm upgrade failed: cannot patch "private" with kind Gateway: admission webhook "validation.istio.io" denied the request: configuration is invalid: server cannot have TLS settings for non HTTPS/TLS ports

Versions

• Bigbang: 2.5.0
• Istio: 1.17.3-bb.1
• K8s: v1.24.16-eks-2d98532

Context

We attempted to expose a custom TCP port on our Gateway with the following BB config snippet:

istio:
  gateways:
    public:
      hosts:
        - "{{ .Values.domain }}"
        - "*.{{ .Values.domain }}"
    private:
      hosts:
        - "*.{{ .Values.domain }}"
      ports:
        - name: https
          number: 8443
          protocol: HTTPS
        - name: tcp-custom
          number: 7687
          protocol: TCP

Viewing the bigbang/istio-bigbang-values secret shows the following (truncated) config:

gateways:
  private:
    servers:
    - hosts:
        - '*.bigbang.dev'
      port:
        name: https
        number: 8443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: private-cert
    - hosts:
        - '*.bigbang.dev'
      port:
        name: tcp-custom
        number: 7687
        protocol: TCP
      tls:
        mode: SIMPLE
        credentialName: private-cert

Reconciling the istio helm release results in the error shown above. It appears the tls section of the server host entry is added regardless of the port protocol resulting in the invalid configuration.

Temporary Workaround

As a stopgap solution, we were able to override the gateway's servers via istio.values:

istio:
  values:
    gateways:
      private:
        servers:
        - hosts:
            - '*.bigbang.dev'
          port:
            name: https
            number: 8443
            protocol: HTTPS
          tls:
            mode: SIMPLE
            credentialName: private-cert
        - hosts:
            - '*.bigbang.dev'
          port:
            name: tcp-custom
            number: 7687
            protocol: TCP
          # NOTE WE HAVE EXCLUDED THE TLS CONFIG FROM THIS HOST

It is also worth noting that when adding custom ports, 8443 or some other HTTPS port must also be included as shown above. Otherwise the helm upgrade fails similarly with:

Upgrade "istio-system-istio" failed: cannot patch "private" with kind Gateway: admission webhook "validation.istio.io" denied the request: configuration is invalid: server config must contain at least one host

Recommended Solutions

• Wrap the tls configuration block in a conditional based on port protocol
• Adjust the documented example to include (or explain the need for) the HTTPS port along with custom additions

[p1-repo-sync-bot[bot]]

Issue 'TLS configuration is invalid with custom TCP ports' closed from GitLab side

[p1-repo-sync-bot[bot]]

wyatt.fry commented:

---

I tried installing the BB helm chart on master branch with these values:

istio:
  gateways:
    public:
      ports:
        - name: https
          number: 8443
          protocol: HTTPS
        - name: tcp-custom-will-it-break
          number: 7687
          protocol: TCP

And it succeeded, despite having a non HTTPS port that the chart gives a tls property. For thoroughness, I installed once and then upgraded twice.

As I was unable to reproduce the problem, I will close this issue.