search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.53k stars 8.25k forks source link

nginx-ingress controller not using the correct TLS cert #4674

Closed cto-bondhouse closed 4 years ago

cto-bondhouse commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

NGINX Ingress controller version:

0.25.1

Kubernetes version (use kubectl version):

v1.13.10-eks-5ac0f1

Environment:

What happened:

Despite having a secret matching the spec.tls.host[].secretName property in my Ingress resource, I am still seeing the Fake Kubernetes Certificate being used when I visit my site

Ingress Resource Configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: all-ingress
  annotations:
    cert-manager.io/issuer: "letsencrypt-prod"
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/auth-url: "https://$host/iam/api/v1/_authorize"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/iam/api/v1/login?originalRequestUri=$escaped_request_uri"
spec:
  tls:
    - hosts:
        - redacted.xxx.com
      secretName: ingress-tls # this is available in secret and I can decode it
  rules:
    - http:
        paths:
          - path: /myapp
            backend:
              serviceName: myapp
              servicePort: 5000

I can validate that the secret exists:

echo $(kubectl get secret ingress-tls -o jsonpath='{.data.tls\.crt}') | base64 -D | openssl x509 -text -noout

# produces the correct TLS certificate & validated

Even the controller logs didn't complain (I restarted the pod several times to see that'd pick up the change) NGINX Ingress Controller Logs:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.25.1
  Build:         git-5179893a9
  Repository:    https://github.com/kubernetes/ingress-nginx/
  nginx version: openresty/1.15.8.1

-------------------------------------------------------------------------------

I1013 12:18:04.256261       6 flags.go:192] Watching for Ingress class: nginx
W1013 12:18:04.256632       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: openresty/1.15.8.1
W1013 12:18:04.260377       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1013 12:18:04.260612       6 main.go:183] Creating API client for https://10.100.0.1:443
I1013 12:18:04.301666       6 main.go:227] Running in Kubernetes cluster version v1.13+ (v1.13.10-eks-5ac0f1) - git (clean) commit 5ac0f1d9ab2c254ea2b0ce3534fd72932094c6e1 - platform linux/amd64
I1013 12:18:04.313247       6 main.go:91] Validated default/default as the default backend.
I1013 12:18:04.556281       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W1013 12:18:04.557816       6 main.go:106] Using deprecated "k8s.io/api/extensions/v1beta1" package because Kubernetes version is < v1.14.0
I1013 12:18:04.615111       6 nginx.go:274] Starting NGINX Ingress controller
I1013 12:18:04.711964       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"default-nginx-ingress-controller", UID:"17036fb6-ed02-11e9-b9f3-0a9a85ba55d0", APIVersion:"v1", ResourceVersion:"12419309", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/default-nginx-ingress-controller
I1013 12:18:05.725152       6 backend_ssl.go:66] Adding Secret "default/ingress-tls" to the local store
I1013 12:18:05.727305       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"all-ingress", UID:"439823ab-c3b9-11e9-97e1-121e92359dda", APIVersion:"extensions/v1beta1", ResourceVersion:"12419301", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/all-ingress
I1013 12:18:05.727429       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"iam", UID:"13027816-c3b5-11e9-97e1-121e92359dda", APIVersion:"extensions/v1beta1", ResourceVersion:"12419312", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/iam
I1013 12:18:05.815793       6 nginx.go:318] Starting NGINX process
I1013 12:18:05.816113       6 leaderelection.go:235] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
I1013 12:18:05.817590       6 controller.go:133] Configuration changes detected, backend reload required.
I1013 12:18:05.820027       6 status.go:86] new leader elected: default-nginx-ingress-controller-fb4fd6546-svccv
I1013 12:18:06.020933       6 controller.go:149] Backend successfully reloaded.
I1013 12:18:06.021102       6 controller.go:158] Initial sync, sleeping for 1 second.
192.168.191.214 - [192.168.191.214] - - [13/Oct/2019:12:18:09 +0000] "GET /metrics HTTP/1.1" 400 261 "-" "Prometheus/2.12.0" 246 0.000 [] [] - - - - 3bd9136a61f69ed2110356fd240b2ebc
192.168.191.214 - [192.168.191.214] - - [13/Oct/2019:12:18:24 +0000] "GET /metrics HTTP/1.1" 400 261 "-" "Prometheus/2.12.0" 246 0.000 [] [] - - - - 125afc687a5f35ce00d9f8c03d9937d3
192.168.191.214 - [192.168.191.214] - - [13/Oct/2019:12:18:39 +0000] "GET /metrics HTTP/1.1" 400 261 "-" "Prometheus/2.12.0" 246 0.000 [] [] - - - - 6ccc16a89decbea92d5a9886eafd28f4
192.168.191.214 - [192.168.191.214] - - [13/Oct/2019:12:18:54 +0000] "GET /metrics HTTP/1.1" 400 261 "-" "Prometheus/2.12.0" 246 0.000 [] [] - - - - fafb0fb86f2d898c4e2be291b4863e04

What you expected to happen:

The correct TLS certificate to be used (referenced by the secret name)

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

Here is the Helm Installed nginx ingress deployment:

Name:                   default-nginx-ingress-controller
Namespace:              default
CreationTimestamp:      Sat, 12 Oct 2019 09:55:46 -0400
Labels:                 app=nginx-ingress
                        chart=nginx-ingress-1.17.0
                        component=controller
                        heritage=Tiller
                        release=default
Annotations:            deployment.kubernetes.io/revision: 1
                        field.cattle.io/publicEndpoints:
                          [{"addresses":["afd91eab9ecf711e9b9f30a9a85ba55d-2045775651.us-east-1.elb.amazonaws.com"],"port":80,"protocol":"TCP","serviceName":"defaul...
Selector:               app=nginx-ingress,component=controller,release=default
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app=nginx-ingress
                    component=controller
                    release=default
  Service Account:  default-nginx-ingress
  Containers:
   nginx-ingress-controller:
    Image:       quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1
    Ports:       80/TCP, 443/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=default/default
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=default/default-nginx-ingress-controller
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   default-nginx-ingress-controller-fb4fd6546 (1/1 replicas created)
Events:          <none>
cto-bondhouse commented 5 years ago

Furthermore, when I shell into the pod itself, I can only see the default fake certificate:

www-data@default-nginx-ingress-controller-fb4fd6546-2cz62:/etc/ingress-controller/ssl$ ls
default-fake-certificate.pem

Even grep-ing the conf doesn't reveal the correct certificate is stored anywhere:

www-data@default-nginx-ingress-controller-fb4fd6546-2cz62:/etc/nginx$ cat nginx.conf | grep ssl
                        is_ssl_passthrough_enabled = false,
                listen_ports = { ssl_proxy = "442", https = "443" },
        ssl_protocols TLSv1.2;
        ssl_session_cache builtin:1000 shared:SSL:10m;
        ssl_session_timeout 10m;
        # allow configuring ssl session tickets
        ssl_session_tickets on;
        ssl_buffer_size 4k;
        # allow configuring custom ssl ciphers
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDH
E-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;
        ssl_ecdh_curve auto;
        proxy_ssl_session_reuse on;
                listen 443 default_server reuseport backlog=511 ssl http2 ;
                ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
                ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;
                ssl_certificate_by_lua_block {
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                        proxy_ssl_server_name       on;
                                        force_ssl_redirect = true,
                                        force_ssl_redirect = true,
                                        force_ssl_redirect = false,
aledbf commented 5 years ago

@erfang-bondhouse how are you testing this?

aledbf commented 5 years ago

Furthermore, when I shell into the pod itself, I can only see the default fake certificate:

This is expected. The SSL certificates are handled by lua. Only the fake certificate is generated and stored in a file.

cto-bondhouse commented 5 years ago

@erfang-bondhouse how are you testing this?

Using any client, in my case mobile Safari and Chrome, it's clear that the fake certificate was being served up

EDIT: a bit of background is that, I first uninstalled nginx ingress controller via helm del --purge and then reinstalled it with a new default backend via:

helm install \
--set controller.defaultBackendService=default/default \
--set defaultBackend.enabled=false stable/nginx-ingress \
--name default
cto-bondhouse commented 5 years ago

fyi the work around is: --default-ssl-certificate=default/ingress-tls which is not a great solution as the cluster could host multiple TLS certificates provisioned separately

I am confused as this seems to be a regression from before I helm reinstalled it from stable/nginx-ingress

I see a lot of release notes related to dynamic TLS improvements between 0.24 and 0.26, is it possible that this is a known bug already?

mogu1986 commented 5 years ago

So do I. the solution is -- default SSL certificate.

goodidea-kp commented 5 years ago

proposed work around "--default-ssl-certificate=default/ingress-tls" doesn't work for me. chart=nginx-ingress-1.24.3

grv231 commented 5 years ago

Quick confirmation, I am running into the same problem, but in my case, I am trying to use ACM (Amazon certificates) and using the annotation service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn.xxx....."

However, when I see the logs, I see default certificate created - SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem . Is there a way I can use the ACM certificates instead of the default ones i see in the ingress nginx logs? Trying to make this work on the latest Ingress Nginx - 0.26.1 image

cto-bondhouse commented 5 years ago

Seems like this is a huge regression that many people are experiencing

grv231 commented 5 years ago

It seems I am missing something. I just put a random app with the config which I mentioned in the above post. Again, the log on the backend (ingress nginx pod) shows the Default certificate created. But when I check the ingress certificates of the app (browser), it does show me Amazon ACM ones. The app opens normally as well and http --> https redirect happens (expected behavior). The question is, the default certificates, is that the expected behavior of Lua in the new nginx image?

haroldpadilla commented 4 years ago

I was having the issue op had on this version:

NGINX Ingress controller
  Release:       0.26.1
  Build:         git-2de5a893a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: openresty/1.15.8.2

I just realized that I was missing the host in the rule per se (not sure if this is required, but it fixed the issues and now the cert. being used is the one I'm providing and not the Fake Kubernetes one). Example of my ingress:

spec:
  tls:
  - hosts:
    - domain.com
    secretName: letsencrypt-staging
  rules:
  - **host: domain.com**
     http:
      paths:
      - path: /
        backend:
          serviceName: "name"
          servicePort: ###
aledbf commented 4 years ago

Closing. NGINX utilizes SNI to determine which SSL Certificate should be used http://nginx.org/en/docs/http/configuring_https_servers.html#name_based_https_servers When an ingress without a host is defined, the default server (_ in nginx) is used. This applies also to connections without SNI support. In these cases, the default SSL certificate is the only way to provide a valid SSL certificate.

Also, if an ingress without a host is defined, the SSL certificate defined in the tls secretName section cannot be used (nginx doesn't know how to use that because of the lack of hostname)

Ayanrocks commented 4 years ago

I was having the issue op had on this version:

NGINX Ingress controller
  Release:       0.26.1
  Build:         git-2de5a893a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: openresty/1.15.8.2

I just realized that I was missing the host in the rule per se (not sure if this is required, but it fixed the issues and now the cert. being used is the one I'm providing and not the Fake Kubernetes one). Example of my ingress:

spec:
  tls:
  - hosts:
    - domain.com
    secretName: letsencrypt-staging
  rules:
  - **host: domain.com**
     http:
      paths:
      - path: /
        backend:
          serviceName: "name"
          servicePort: ###

your comment saved my life. Working on this for 17 hours straight and finally a single line solved it. Thank you so much.

mac2000 commented 4 years ago

Not sure if this issue is a right place but wish to comment it instead of creating new one, somehow ingress controller does not use provided wildcard tls and uses "Kubernetes Ingress Controller Fake Certificate" instead for requests without SNI

e.g.:

---
apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: demo
  labels:
    app: demo
spec:
  containers:
    - name: demo
      image: nginx:alpine
      ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: demo
spec:
  type: NodePort
  ports:
    - name: demo
      port: 80
      protocol: TCP
  selector:
    app: demo
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: default
  name: demo
spec:
  tls:
    - hosts:
        - "demo.dev.contoso.com"
      secretName: dev.contoso.com
  rules:
    - host: "demo.dev.contoso.com"
      http:
        paths:
          - path: /
            backend:
              serviceName: demo
              servicePort: demo

when using curl -v https://demo.dev.contoso.com/ is see following:

* Server certificate: *.dev.contoso.com
* Server certificate: Sectigo RSA Domain Validation Secure Server CA
* Server certificate: USERTrust RSA Certification Authority

but trying to use openssl s_client -connect demo.dev.contoso.com:443 (without server name):

Certificate chain
 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
   i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate

just for reference calling openssl s_client -servername demo.dev.contoso.com -connect demo.dev.contoso.com:443 gives:

Certificate chain
 0 s:/CN=*.dev.contoso.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA

somehow ingress does not use given wildcard certificate and at the same time, the same config works ok in GKE

aledbf commented 4 years ago

somehow ingress does not use given wildcard certificate and at the same time, the same config works ok in GKE

Check my comment. This is related to SNI https://github.com/kubernetes/ingress-nginx/issues/4674#issuecomment-569493421 You can use set a different default SSL certificate adding the flag --default-ssl-certificate in the ingress controller deployment

mac2000 commented 4 years ago

@aledbf sorry I did not mention this, default certificate on ingress will not solve this - in my case there is multiple domains, e.g.: .dev.contoso.com, .test.contoso.com, *.staging.contoso.com, etc

Indeed we can put default cert but it will work only if we are talking about single domain per cluster in my case error will be the same with only difference that instead of fake certificate error will be on wrong CN

aledbf commented 4 years ago

@mac2000 when a client without SNI support sends a request it will be redirected to a default nginx server block. In there you can have one SSL certificate. This acts as a fallback.

mac2000 commented 4 years ago

@aledbf yep it sounds reasonable but then "default backend" in ingress config is useless in this case, wondering how it does work in GKE (probably it is because of uniq ip addresses per ingress)

at the very end there is still an issue, seems like there is no easy way to have multiple wildcard certificates attached to kubernetes (an easy way will be to have multiple clusters 🤔)

aledbf commented 4 years ago

@mac2000 the GKE controller create separated instances per ingress definition. Is not comparable with ingress-nginx where we share the definitions by default. You can have one deployment per wildcard domain, using the ingress.class annotation, sharing N definitions for a subdomain like *.dev.contoso.com, setting the default SSL certificate there. Each deployment will create a different LB.

aledbf commented 4 years ago

@mac2000 default backend != default SSL certificate. What I've been mentioning is a flag in the deployment, not related to the default backend.

mac2000 commented 4 years ago

Thank you for clarification, I went here from microk8s which just gives us just microk8s enable ingress, if I understand your point correct - we gonna need to deploy multiple ingress controllers (each with its own wildcard cert set as default one and bound to separate ip address) and then in our application deployments use correct ingress controller or as mentioned before having multiple clusters per wildcard domain

aledbf commented 4 years ago

we gonna need to deploy multiple ingress controllers

@mac2000 only if you have clients without SNI support.

CameronHudson8 commented 3 years ago

I'd also like to add that for HTTP requests, you can set the SNI by adding the header host: example.com.

pcgeek86 commented 2 years ago

I'm having the same problem with my NGINX ingress controller. It's serving up the default FAKE certificate instead of the one that was issued by cert-manager and stored in a secret. I have configured the secretName property.

Why is this issue closed? How are people handling this years later?

AATHITH commented 2 years ago

I have used the TLS secret which is not present in the ArgoCD namespace. Ingress used the default fake kubernetes certificate(was expecting an error stating "secret not found").

Mushfik commented 2 years ago

Is this issue fixed? I am still facing the same issue in ingress-nginx v1.2.0. After adding spec.tls.Hosts, spec.tls.secretName (CA Cert) and spec.rule.Host, it is still using default fake kubernetes certificate.

PhotoTeeborChoka commented 2 years ago

Encountering this as well. Everything is defined, yet the certificate served for a wildcarded host is the default fake certificate. How did you manage to solve this issue?

mrj0 commented 1 year ago

I encountered this - it turns out that wildcard host names are supported in the rules section but not by SNI, so it serves the fake ingress cert instead of mine. I had to write out 8 rule blocks instead of the wildcard.

mahendranbhathp commented 1 year ago

I encountered this - it turns out that wildcard host names are supported in the rules section but not by SNI, so it serves the fake ingress cert instead of mine. I had to write out 8 rule blocks instead of the wildcard.

could you please elobertae more? I am also facing same issue!

redebeer commented 1 year ago

After countless hours of head scratching and puzzling, I came across something that appears to have worked for me. In my ingress, under spec change the entry under hosts eg from - example.domain.com to - ".domain.com". Source for my fix: https://docs.digitalocean.com/tutorials/set-up-nginx-ingress-controller-to-use-wildcard-certificates/#:~:text=%2D%20hosts%3A-,%2D%20%22.doks%2Dstarter%2Dkit.page%22,-secretName%3A%20doks

sridharakb05 commented 9 months ago

I am running into the same problem, but in my case, I am trying to use ACM (Amazon certificates) and using the annotation service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn.xxx....."

You need to uninstall and install the ingress controller again with the ACM.