ssl-passthrough returning ingress cert

[paulbdavis]

Is this a request for help? Yes

What keywords did you search in NGINX Ingress controller issues before filing this one? grpc ssl-passthrough ingress

---

Is this a BUG REPORT or FEATURE REQUEST? bug?

NGINX Ingress controller version: 0.10.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.11-gke.1", GitCommit:"3500f53730c1fea7b57901977df165c3eb317bce", GitTreeState:"clean", BuildDate:"2017-12-08T18:05:07Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GKE

What happened: openssl s_client returns cert default ingress cert

What you expected to happen: openssl s_client returns cert from application

How to reproduce it (as minimally and precisely as possible):

Ingress controller, there is another ingress that does not use ssl-passthrough that comes after this one, there is no overlap in the hosts

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grpc-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: rpc.host.tld
    http:
      paths:
      - backend:
          serviceName: svc-grpc
          servicePort: 443
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway-ingress
  annotations:
    # set for letsencrypt support
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - api.host.tld
    secretName: api-tls
  rules:
  - host: api.host.tld
    http:
      paths:
      - backend:
          serviceName: gateway
          servicePort: 8080

Service for this. The application's gRPC server is listening on 443 as well (previosuly this service forwarded 443 to 10000, changed the gRPC listening port to tes if that was the issue)

apiVersion: v1
kind: Service
metadata:
  name: svc-grpc
  labels: ...
spec:
  ports:
  - name: grpc-proxy-port
    port: 443
    targetPort: 443
  selector: ...

To test that the application and service were set up right, I set the service to type: LoadBalancer and ran openssl s_client against the service's IP directly and the correct cert was returned

[paulbdavis]

I set up basically the same way as the guy from this issue, but I am still getting the Kubernetes Ingress Controller Fake Certificate

I also had to add the kubernetes.io/ingress.class: "nginx" annotation, otherwise it got picked up as a GCE ingress.

[aledbf]

@paulbdavis do you have the flag --enable-ssl-passthrough in the nginx ingress controller deployment?

[aledbf]

@paulbdavis are you using openssl s_client -servername rpc.host.tld?

[paulbdavis]

Yes, --enable-ssl-passthrough is on the ingress controller

script I am using to check the cert

cat ~/bin/check-cert 
#!/usr/bin/env bash

echo | openssl s_client -showcerts -servername $1 -connect $1:${2:-443} 2>/dev/null | openssl x509 -inform pem -noout -text

[paulbdavis]

This is the config for the controller itself

      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.10.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --enable-ssl-passthrough

[paulbdavis]

Interestingly, kubectl reports that there is only port 80 open on the passthrough ingress

kubectl get ing
NAME                HOSTS           ADDRESS   PORTS     AGE
gateway-ingress     api.host.tld    x.x.x.x   80, 443   16m
grpc-ingress        rpc.host.tld    x.x.x.x   80        17m

[dghubble]

I'm running nginx-ingress 0.10.0 with --enable-ssl-passthrough and a service annotated in the same way, and getting the "Kubernetes Ingress Controller Fake Certificate" as well (via openssl s_client -connect). I also see the same port quirk showing port 80 for the endpoint that should be 443 and passthrough.

kubectl get ingress
NAME            HOSTS                             PORTS     AGE
matchbox        matchbox.domain.com               80        2s
matchbox-rpc    matchbox-rpc.domain.com           80        2s  <- passthrough, should be 443

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: matchbox-rpc
  annotations:
    kubernetes.io/ingress.class: "public"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
    - host: matchbox-rpc.domain.com
      http:
        paths:
          - path: /
            backend:
              serviceName: matchbox
              servicePort: 8081

I've also tried adding a fake tls section to the spec as this was sometimes required in the past. No dice.

Generated nginx.conf snippet:

   ## start server matchbox-rpc.domain.com
    server {
        server_name matchbox-rpc.domain.com ;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        location / {
            port_in_redirect off;

            set $proxy_upstream_name "default-matchbox-8081";

            set $namespace      "default";
            set $ingress_name   "matchbox-rpc";
            set $service_name   "matchbox";

[maennchen]

I have the exact same problem. It seem though as passthrough is not working at all. Besides the Fake certificate I also get the default backend while I get the correct backend with HTTP.

(--enable-ssl-passthrough is enabled, nginx.ingress.kubernetes.io/ssl-passthrough: "true" is set)

[paulbdavis]

Eager to try this out and see if it was fixed by #1945

Is there an image I can use to test it or would I need to build it myself?

[aledbf]

@paulbdavis you can use quay.io/aledbf/nginx-ingress-controller:0.317

[paulbdavis]

All issues fixed, thanks for the quick response

[sands6]

SSL passhrough with nginx-ingress-controller:0.10.1 does not fix issue. Please reopen this issue. I am still getting the Kubernetes Ingress Controller Fake Certificate for tls passthrough use case, and ingress controller is terminating TLS at Ingress controller and sending http request to pod. Pod responds with 400 error saying expecting TLS connection

[sands6]

Here is the ingress.conf for tls pass through:

start server nginx-tls-svc.apps.sbox.k8s.com server { server_name nginx-tls-svc.apps.sbox.k8s.com ;

    listen 80;

    listen [::]:80;

    set $proxy_upstream_name "-";

    listen 442 proxy_protocol   ssl http2;

    listen [::]:442 proxy_protocol  ssl http2;

    # PEM sha: f0150e56d054b0ac627d371ff746d95df9bb4a39
    ssl_certificate                         /ingress-controller/ssl/default-                                                   fake-certificate.pem;
    ssl_certificate_key                     /ingress-controller/ssl/default-                                                   fake-certificate.pem;

    more_set_headers                        "Strict-Transport-Security: max-                                                   age=15724800; includeSubDomains;";

See the fake certificate being passed in to ingress controller

[sands6]

openssl s_client -showcerts -servername nginx-tls.svc.apps.sbox.k8s.com -connect tlsterm.apps.sbox.k8s.com:32005 2>/dev/null

CONNECTED(00000003)

Certificate chain 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate -----BEGIN CERTIFICATE----- MIIDcDCCAligAwIBAgIRAJjQZ365z39vtnrGsapkR2wwDQYJKoZIhvcNAQELBQAw SzEQMA4GA1UEChMHQWNtZSBDbzE3MDUGA1UEAxMuS3ViZXJuZXRlcyBJbmdyZXNz IENvbnRyb2xsZXIgRmFrZSBDZXJ0aWZpY2F0ZTAeFw0xODAxMjUyMTIzMjhaFw0x OTAxMjUyMTIzMjhaMEsxEDAOBgNVBAoTB0FjbWUgQ28xNzA1BgNVBAMTLkt1YmVy bmV0ZXMgSW5ncmVzcyBDb250cm9sbGVyIEZha2UgQ2VydGlmaWNhdGUwggEiMA0G CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/8v6wZ3lVSOkNETGOMzKa143idn7/ wf1zJNUDMaqrzEdxgGtAVIz0lSNXIA+lsghyOwVGh0TIS+1n35cMiMTS1LyRQG6r b7AbZo6EdIhjKNgxEoFGBkN3Cj8OgAOIPmmpyn5BFg9lzxeUQ+bmbQP4+ri+aA/U ujWvRJwcCto4obn2P8B5flQs6S1O1WI3EfxVdBU/x4WdB70F6v2MpgYoxDNcXFcj LBI8+2grN13Qc/27mFiDSYfbwsWgM1wPXhfTGDaiHTTJEO6OEE7T2+LbPIBS7r+j ipUDrqyPntewrF59byy9dPjZEQn8MP5lb2+OdCE2mVUjo+3bm4lEWptdAgMBAAGj TzBNMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB Af8EAjAAMBgGA1UdEQQRMA+CDWluZ3Jlc3MubG9jYWwwDQYJKoZIhvcNAQELBQAD ggEBABFe+i/8gtZNWz1C6KENheWeG1wIebWaKMOfAPsRY6STDzQI/IbwbfJPd9gO zUxDiuW8z9po1k0zyUEHJ+UMYEhbCsQ8m7obT8fitPKhbkEEjy6kU7sygOwxLTN1 Tz9zDvoXjvzQube1DaVrynHszuBNYIhB5iqMyUhWLneOOPe/YFQfiNJztU/5x5gb JROc7ECu58co7MTb+lH+2nyYhG18K0i7jel7bre62no1DE2DTbLcqjqodIbi+0JL kRJzLfi+qXG/A1Pxp2+5URnDwGEtHx90AaHGVm8ioNrDil08SN5SYKFOq7CuJdqv k2lSSJOu+wcBmtCshJMb/B8iRA4= END CERTIFICATE

Server certificate subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate

No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 1602 bytes and written 476 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 9C12424DF652782636622065FA452FB7B6812D7F843B80ED231839ADB276C32C Session-ID-ctx: Master-Key: CE3C49591CBEA6D7683C51FF4B8D42AE13F98E5DAF0EE5389DBD66694342410946918A72702A2CC904B8B9279A3F6EB2 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 600 (seconds) TLS session ticket: 0000 - 1c 2e 87 71 2b 1d 80 5f-87 94 82 7e 85 14 77 a5 ...q+.._...~..w. 0010 - b0 66 a2 3f 10 37 48 39-25 26 50 9a e1 63 89 33 .f.?.7H9%&P..c.3 0020 - 6a d2 b9 ee 84 f4 6c ac-d8 bb 0c 1e 80 06 fa 49 j.....l........I 0030 - fe 7b 7a 8e 88 ec 1a e8-06 27 e5 09 de 10 76 a3 .{z......'....v. 0040 - d5 16 83 ef ba 93 f1 3d-77 81 4d fb 71 64 2b 6a .......=w.M.qd+j 0050 - 81 0d 95 40 34 1b 90 e6-11 67 9e 1b 6d 78 ef 01 ...@4....g..mx.. 0060 - e6 89 c5 48 8c 5f 51 d7-f6 ca 0a 07 52 25 27 47 ...H._Q.....R%'G 0070 - 5c 83 4d b4 e9 10 b8 c7-8c cf 97 7d 9b ef c4 4d .M........}...M 0080 - cb 13 29 27 33 ca cf c2-8c 6c 36 50 19 0c 03 c0 ..)'3....l6P.... 0090 - d2 c7 eb ab 78 60 f9 13-78 b6 e9 c4 9d 33 0d 6e ....x..x....3.n 00a0 - 50 2b 78 e8 31 f3 bd 26-63 57 ad 14 95 69 98 60 P+x.1..&cW...i. 00b0 - 2c 48 d4 82 0a 3c 56 8e-8d 30 b6 22 82 10 95 57 ,H...<V..0."...W 00c0 - ae 3d dc 4b 08 59 72 9e-b5 d1 37 61 d1 03 6c 59 .=.K.Yr...7a..lY 00d0 - ba 0b 70 a7 6d 5e 1d 50-e1 b6 82 e5 1d 85 45 64 ..p.m^.P......Ed

Start Time: 1516977899
Timeout   : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)

[sands6]

reopen #1945

[dghubble]

It's solved when I use 0.10.1.

[sands6]

@dghubble Could you share your ingress manifest? I am running into problems as you ran previously with 0.10.0. Tls termination works perfectly but with TLS passthrough I am routed to default backend kubectl get ingress -n examples NAME HOSTS ADDRESS PORTS AGE nginx-term tlsterm.apps.sbox.k8s.com 80, 443 1d nginx-tls nginx-tls-svc.apps.sbox.k8s.com 80 55m

curl -k https://nginx-tls-svc.apps.sbox.k8s.com default backend - 404

[sands6]

I have tested TLS passthrough with 0.10.1 and 0.10.2 versions and TLS passthrough does not work. Although TLS passthrough works with 0.9.0-beta.17 version. I am 100% certain that #1947 is not fixed with 0.10.1 and 0.10.2. Please reopen #1947

[dghubble]

Following the update to 0.10.1 (and now on 0.10.2), the ingress listing is correct compared with before. Pass-through is working for me:

matchbox        matchbox.domain.com                 80        2d     <- HTTP
matchbox-rpc    matchbox-rpc.domain.com             80, 443   2d   <- ssl-passthrough

Maybe revisit the ingress resources themselves.

[paulbdavis]

This image worked for the specific problem I was having

quay.io/aledbf/nginx-ingress-controller:0.317

I have not had a chance to test the official 0.10.x images yet though, so I am not sure if they fix it. @sands6 maybe try that and see if it works.

Also, I noticed that the passthrough only works if it is a proper SNI request, otherwise the default ingress cert is returned.

[lander2k2]

I'm seeing similar behavior to @sands6. Works on version 0.9.0-beta.17 but broken on 0.10.1.

On version 0.10.1 I get a 400 error on the backend: The plain HTTP request was sent to HTTPS port and the default ingress cert is returned to the client.

Environment: AWS

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:59:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4+coreos.0", GitCommit:"4292f9682595afddbb4f8b1483673449c74f9619", GitTreeState:"clean", BuildDate:"2017-11-21T17:22:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Ingress controller args:

      containers:
        - name: nginx-ingress-lb
          #image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.10.1
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17
          args:
          - /nginx-ingress-controller
          - --configmap=$(POD_NAMESPACE)/ingress-controller-config
          - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
          - --publish-service=$(POD_NAMESPACE)/test-lb
          - --enable-ssl-passthrough

Ingress object:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  namespace: test
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
  - hosts:
    - nginx.example.com
  rules:
  - host: nginx.example.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 443

Backend nginx deployment, service:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: test
  labels:
    app: nginx
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app: nginx
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx
  namespace: test
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: nginx:1.13.8
        imagePullPolicy: "IfNotPresent"
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        livenessProbe:
          httpGet:
            path: /index.html
            port: 80
          initialDelaySeconds: 30
          timeoutSeconds: 1
          failureThreshold: 6
        readinessProbe:
          httpGet:
            path: /index.html
            port: 80
          initialDelaySeconds: 30
          timeoutSeconds: 1
          failureThreshold: 6
        resources:
          limits:
            memory: 512Mi
            cpu: 200m
          requests:
            cpu: 100m
            memory: 256Mi
        volumeMounts:
        - mountPath: /etc/nginx/conf.d
          name: configmap
        - mountPath: /etc/nginx/ssl
          name: tls
      volumes:
      - name: configmap
        configMap:
          name: nginx-config
          defaultMode: 0744
      - name: tls
        secret:
          secretName: nginx-tls

[spuranam]

@lander2k2 @sands6 The default annotation prefix has changed between beta-17 and 0.10.x version of the ingress-controller, you change the annotation ingress.kubernetes.io/ssl-passthrough: "true" to nginx.ingress.kubernetes.io/ssl-passthrough: "true". Notice the lead "nginx.". If could not change the annotation then you need to pass this additional cli argument to ingress controller "- --annotations-prefix=ingress.kubernetes.io".

[veeshall]

Changed to 0.9.0-beta.17, and then / ingress.kubernetes.io/ssl-passthrough: "true"/ works fine. Tried lot of other options (including 0.10.0), and none of them worked for me.

[Rajczyk]

When you havenginx.ingress.kubernetes.io/ssl-passthrough: "true" are you supposed to get the message : The plain HTTP request was sent to HTTPS port. When you go to the http://domain:443? is there a way to turn this off?

[zhjchen]

0.12.0 works fine. However I have some nginx logging issue. there is no event logged when I do curl/query our service behind nginx controller.

https://github.com/kubernetes/ingress-nginx/issues/2329

[aliostad]

I found out you need to add annotation

nginx.ingress.kubernetes.io/secure-backends: "true"

[opskumu]

Secure backends DEPRECATED (since 0.18.0) nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" replaced, and removed 0.21.0

https://github.com/kubernetes/ingress-nginx/blob/nginx-0.20.0/docs/user-guide/nginx-configuration/annotations.md#secure-backends-deprecated-since-0180

[riker09]

It feels like I'm running into this issue.

Here's my (non-working) ingress.yaml:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/server-alias: ".domain.tld"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: "domain.tld"
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: 443

The weird thing is, when I remove the server-alias and instead define the subdomains individually I'm getting the correct certificate that is presented by my-service. I have tried to remove the forced (or unforced) SSL redirect that happens automagically (see commented nginx.ingress.kubernetes.io/force-ssl-redirect: "false") but to no avail.

[EDIT] I have double-checked and verified that the nginx ingress controller has the --enable-ssl-passthrough flag. The second part would not work otherwise (as to my understanding, anyway) [/EDIT]]

So, this is a working configuration (certificate is coming from the my-service):

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    # nginx.ingress.kubernetes.io/server-alias: ".domain.tld"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: "sub.domain.tld"
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: 443

But since I have many subdomains I would rather not have to list each of them in the ingress. This would be a nightmare to maintain.

[EDIT #2] I'm using RKE to provision my cluster and the Docker image for the nginx ingress controller is rancher/nginx-ingress-controller:nginx-0.25.1-rancher1. It looks like the used version is some month behind official releases? [/EDIT]

[EDIT #3] I have removed the RKE nginx-ingress controller and have it replaced with the official helm chart stable/nginx-ingress. Yet the results are the same. [/EDIT]

[tacerus]

Hi, have you ever figured this out?

[riker09]

@tacerus If that question was directed at me then I have to disappoint you. I have abandoned this ingress a long time ago.

[tacerus]

I suppose it was, but my setup has since magically started to work fine! Thank you for the reply!

[chancez]

I managed to get it working for using a gRPC backend service, but I had to configure TLS on the ingress and set the backend-protocol to GRPCS, despite the documentation clearly stating that other annotations have no effect when ssl-passthrough is enabled.

[DrVaskee]

I'm sorry to reply to this old issue. But i was fighting with this the last few days and maybe this might help someone that stumbles across this issue.

When configuring an ingress object you have to specify your hostnames as lowercase. The Ingress-Objects enforces this. But wenn you try e.g. with the openssl s_client it matters how you would sent the -servername

For example:

openssl s_client -connect sub.domain.tld:443 -servername sub.domain.tld -showcerts --> works and gives the correct certificate configured in your backend

openssl s_client -connect sub.domain.tld:443 -servername sub.DOMAIN.tld -showcerts --> returns the Kubernetes Fake Certificate

---

Also a question if someone can answer it: Is this intended behaviour or a bug?