client <-> gateway <-> backend mTLS fail.

[zzjin]

At our multi user cluster, we want to use mtls in client to gateway and gateway to backend service both to provide Zero Trust Mesh service.

1. Ingress yaml:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
   annotations:
   # change to https
   nginx.ingress.kubernetes.io/backend-protocol: HTTPS
   # client to gateway mtls
   nginx.ingress.kubernetes.io/auth-tls-secret: wildcard-cert-cacert
   # gateway to backend mtls
   nginx.ingress.kubernetes.io/proxy-ssl-secret: ns-6qo6sdtp/ztm-cert
   name: network-rhhwxntqnjuo
   namespace: ns-6qo6sdtp
   spec:
   ingressClassName: nginx
   rules:
   - host: juwytrmsxdgk.sealos.io
     http:
       paths:
         - backend:
             service:
               name: ztm-hub
               port:
                 number: 8888
           path: /
           pathType: Prefix
   tls:
   - hosts:
       - juwytrmsxdgk.sealos.io
     secretName: wildcard-cert

2. client mtls cert

   apiVersion: v1
   data:
   cacert: xxxx==
   kind: Secret
   metadata:
   # name must to be ("hosts-secretName"-cacert)
   name: wildcard-cert-cacert
   namespace: ns-6qo6sdtp
   type: Opaque

3. gateway mtls sercret

   apiVersion: v1
   data:
   tls.crt: xxxx-crt
   tls.key: xxxx-key
   kind: Secret
   metadata:
   name: ztm-cert
   namespace: ns-6qo6sdtp
   type: kubernetes.io/tls

But test with curl:

curl -v --cert client.crt --key client.key --cacert ca.crt https://juwytrmsxdgk.sealos.io

with error:

*   Trying 1.2.3.4:443...
* Connected to juwytrmsxdgk.sealos.io (1.2.3.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: ca.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=sealos.io
*  start date: May 26 14:29:55 2024 GMT
*  expire date: Aug 24 14:29:54 2024 GMT
*  subjectAltName: host "juwytrmsxdgk.sealos.io" matched cert's "*.sealos.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55f0f088be90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: juwytrmsxdgk.sealos.io
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 503
< content-length: 159
< content-type: text/plain
< date: Tue, 13 Aug 2024 16:21:08 GMT
< server: istio-envoy
<
* Connection #0 to host juwytrmsxdgk.sealos.io left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: Secret is not supplied by SDS

Related Issue: #1008

[johnlanni]

nginx.ingress.kubernetes.io/proxy-ssl-name: <后端服务域名> nginx.ingress.kubernetes.io/proxy-ssl-server-name: on

加上这两个注解看看呢，怀疑是后端校验SNI，如果不带这两个注解是不带SNI的

[zzjin]

加了也不行.

和应用那边对了下,他们不支持 7 层的 tls termination.