TLS configuration is invalid with custom TCP ports

[p1-repo-sync-bot[bot]]

Overview

When adding custom TCP ports to a gateway definition (as documented here), helm upgrade fails with the following error:

✗ Helm upgrade failed: cannot patch "private" with kind Gateway: admission webhook "validation.istio.io" denied the request: configuration is invalid: server cannot have TLS settings for non HTTPS/TLS ports

Versions

• Bigbang: 2.5.0
• Istio: 1.17.3-bb.1
• K8s: v1.24.16-eks-2d98532

Context

We attempted to expose a custom TCP port on our Gateway with the following BB config snippet:

istio:
  gateways:
    public:
      hosts:
        - "{{ .Values.domain }}"
        - "*.{{ .Values.domain }}"
    private:
      hosts:
        - "*.{{ .Values.domain }}"
      ports:
        - name: https
          number: 8443
          protocol: HTTPS
        - name: tcp-custom
          number: 7687
          protocol: TCP

Viewing the bigbang/istio-bigbang-values secret shows the following (truncated) config:

gateways:
  private:
    servers:
    - hosts:
        - '*.bigbang.dev'
      port:
        name: https
        number: 8443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: private-cert
    - hosts:
        - '*.bigbang.dev'
      port:
        name: tcp-custom
        number: 7687
        protocol: TCP
      tls:
        mode: SIMPLE
        credentialName: private-cert

Reconciling the istio helm release results in the error shown above. It appears the tls section of the server host entry is added regardless of the port protocol resulting in the invalid configuration.

Temporary Workaround

As a stopgap solution, we were able to override the gateway's servers via istio.values:

istio:
  values:
    gateways:
      private:
        servers:
        - hosts:
            - '*.bigbang.dev'
          port:
            name: https
            number: 8443
            protocol: HTTPS
          tls:
            mode: SIMPLE
            credentialName: private-cert
        - hosts:
            - '*.bigbang.dev'
          port:
            name: tcp-custom
            number: 7687
            protocol: TCP
          # NOTE WE HAVE EXCLUDED THE TLS CONFIG FROM THIS HOST

It is also worth noting that when adding custom ports, 8443 or some other HTTPS port must also be included as shown above. Otherwise the helm upgrade fails similarly with:

Upgrade "istio-system-istio" failed: cannot patch "private" with kind Gateway: admission webhook "validation.istio.io" denied the request: configuration is invalid: server config must contain at least one host

Recommended Solutions

• Wrap the tls configuration block in a conditional based on port protocol
• Adjust the documented example to include (or explain the need for) the HTTPS port along with custom additions

[p1-repo-sync-bot[bot]]

montgomery.marcus commented:

---

I tried to reproduce this error by deploying BB with:

helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ./docs/assets/configs/example/policy-overrides-k3d.yaml -f ../overrides/registry-values.yaml -f ./chart/ingress-certs.yaml -f ../overrides/bigbang-1680.yaml

Where bigbang-1680.yaml is the first block of yaml in this issue and I did not see an error. It's possible that this was fixed in istio 1.18.3. From the release notes:

Added ability to install gateway helm chart with a dual-stack service definition.

[p1-repo-sync-bot[bot]]

ablanchard commented:

---

@montgomery.marcus can you share the gateway config that was produced in the updated version? (The equivalent of the second block of yaml in the original post.)

The TLS section still appears to be added regardless of the port protocol in the templated istio values.. I think "dual stack support" is in reference to being able to handle ipv4 and ipv6 simultaneously. :thinking: