SSL Passthrough does not work on AWS NLB

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes

https://kubernetes.github.io/ingress-nginx/

Apache License 2.0

17.51k stars 8.26k forks source link

SSL Passthrough does not work on AWS NLB #9658

Closed ismailyenigul closed 1 year ago

ismailyenigul commented 1 year ago

What happened:

Deployed nginx-ingress controller 4.5.2 on EKS 1.23 with helm chart. Enabled SSL Passthrough. But when I try to access backend configured to run HTTPS only I got

<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>

Backend is running with HTTPS test from inside k8s

$ curl  https://172.20.80.93 -k
test

helm values

ingress-nginx:
  controller:
    extraArgs: {"enable-ssl-passthrough": ""}
    containerPort:
      https: 443
    ingressClass: ingress-kafka
    ingressClassResource:
      name: ingress-kafka
      enabled: true
      default: false
      controllerValue: "k8s.io/ingress-kafka"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-internal: true
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "100"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:region:id:certificate/certid

I can see that SSL Passthrough enabled in ingress controller pod logs

NGINX Ingress controller
  Release:       v1.6.4
  Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

I0221 18:07:04.397300       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0221 18:07:04.467823       7 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0221 18:07:04.490414       7 nginx.go:261] "Starting NGINX Ingress controller"
I0221 18:07:05.792937       7 nginx.go:755] "Starting TLS proxy for SSL Passthrough"
I0221 18:07:05.793027       7 nginx.go:304] "Starting NGINX process"

my ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myservicea
  namespace: kafka
  annotations:
    external-dns.alpha.kubernetes.io/hostname: details.mydomain.com.
    external-dns.alpha.kubernetes.io/ttl: "600"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

spec:
  ingressClassName: ingress-kafka
  rules:
    - host: details.mydomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: details
                port:
                  number: 443

What you expected to happen:

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

Kubernetes version (use kubectl version): 1.23 Environment: AWS

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

longwuyuan commented 1 year ago

/remove-kind bug

I tested the annotation ssl-passthrough on minikube and it works.

Unlike you I did not specify a backend-protocol as it makes no sense.

/close

k8s-ci-robot commented 1 year ago

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9658#issuecomment-1439730311): >/remove-kind bug > >I tested the annotation ssl-passthrough on minikube and it works. > >Unlike you I did not specify a backend-protocol as it makes no sense. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

ismailyenigul commented 1 year ago

/remove-kind bug

I tested the annotation ssl-passthrough on minikube and it works.

Unlike you I did not specify a backend-protocol as it makes no sense.

/close

can you please send your nginx ingress deployment settings and ingress.yaml to test in AWS env. Do you think that AWS NLB issue in my case? Is there any way to test your case in AWS too before closing the ticket?

longwuyuan commented 1 year ago

i just created an ingress with https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#ssl-passthrough

You can do the same and then post the info that is related to the issue directly. Like

kubectl describe
kubectl get svc,ing -A -o wide
kubectl describe ing -A
Curl command in full with -v and its reponse
Complete logs of the controller pod
Any other information

If you find the annotation not working as per documentation, then please re-open the issue.

ismailyenigul commented 1 year ago

@longwuyuan still does not work for AWS NLB There are so many ingress for kafka . here i am mainly testing SSL passthrough over details.mydomain.com ingress and pod. so please dont worry about kafka ones and let's focus on details.mydomain.com only in the following output.

curl output for ingress url:

$ curl -v https://details.mydomain.com/                                                                     
*   Trying 10.0.0.180:443...
* Connected to details.mydomain.com (10.0.0.180) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Aug 14 00:00:00 2022 GMT
*  expire date: Sep 12 23:59:59 2023 GMT
*  subjectAltName: host "details.mydomain.com" matched cert's "*.mydomain.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: details.mydomain.com
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Wed, 22 Feb 2023 15:08:30 GMT
< Content-Type: text/html
< Content-Length: 248
< Connection: close
< 
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):

log for related to curl command:

127.0.0.1 - - [22/Feb/2023:15:18:27 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.86.0" 90 0.000 [] [] - - - - 2c04d78844a6875b835728360db2379e

As you see above, I see the SSL details from AWS certificate, not from the pod which is self signed.

curl test from k8s inside from a pod to details pod

bash-5.1$ curl -v https://10.0.2.151 -k
*   Trying 10.0.2.151:443...
* Connected to 10.0.2.151 (10.0.2.151) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED
*  start date: Dec  5 10:37:17 2015 GMT
*  expire date: Nov 30 10:37:17 2035 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x559e45fa26c0)
> GET / HTTP/2
> Host: 10.0.2.151
> user-agent: curl/7.78.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Wed, 22 Feb 2023 15:21:00 GMT
< content-type: text/html
< content-length: 5
< last-modified: Tue, 21 Feb 2023 16:48:27 GMT
< etag: "63f4f5db-5"
< accept-ranges: bytes
< 
test
* Connection #0 to host 10.0.2.151 left intact

my ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myservicea
  namespace: kafka
  annotations:
    external-dns.alpha.kubernetes.io/hostname: details.mydomain.com.
    external-dns.alpha.kubernetes.io/ttl: "600"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  ingressClassName: ingress-kafka
  rules:
    - host: details.mydomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: details
                port:
                  number: 443

kubectl svc and ing details in kafka namespace


kubectl get svc,ing  -o wide -n kafka                                                                           1866ms

NAME                                                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE   SELECTOR
service/details                                            NodePort       172.20.80.93     <none>                                                                          443:32006/TCP                24h   app=details
service/devbox-cruise-control                              ClusterIP      172.20.106.66    <none>                                                                          9090/TCP                     24h   strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-cruise-control
service/devbox-kafka-bootstrap                             ClusterIP      172.20.53.26     <none>                                                                          9091/TCP,9092/TCP            24h   strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka
service/devbox-kafka-brokers                               ClusterIP      None             <none>                                                                          9090/TCP,9091/TCP,9092/TCP   24h   strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka
service/devbox-kafka-tls-0                                 ClusterIP      172.20.110.50    <none>                                                                          9094/TCP                     23h   statefulset.kubernetes.io/pod-name=devbox-kafka-0,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka
service/devbox-kafka-tls-1                                 ClusterIP      172.20.197.234   <none>                                                                          9094/TCP                     23h   statefulset.kubernetes.io/pod-name=devbox-kafka-1,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka
service/devbox-kafka-tls-2                                 ClusterIP      172.20.243.234   <none>                                                                          9094/TCP                     23h   statefulset.kubernetes.io/pod-name=devbox-kafka-2,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka
service/devbox-kafka-tls-bootstrap                         ClusterIP      172.20.10.227    <none>                                                                          9094/TCP                     23h   strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka
service/devbox-zookeeper-client                            ClusterIP      172.20.3.146     <none>                                                                          2181/TCP                     24h   strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeper
service/devbox-zookeeper-nodes                             ClusterIP      None             <none>                                                                          2181/TCP,2888/TCP,3888/TCP   24h   strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeper
service/kafka-ingress-ingress-nginx-controller             LoadBalancer   172.20.38.32     a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80:32352/TCP,443:31567/TCP   21h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx
service/kafka-ingress-ingress-nginx-controller-admission   ClusterIP      172.20.5.5       <none>                                                                          443/TCP                      21h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx

NAME                                                   CLASS           HOSTS                         ADDRESS                                                                         PORTS     AGE
ingress.networking.k8s.io/devbox-kafka-tls-0           ingress-kafka   kafka-b0.mydomain.com   a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80, 443   23h
ingress.networking.k8s.io/devbox-kafka-tls-1           ingress-kafka   kafka-b1.mydomain.com   a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80, 443   23h
ingress.networking.k8s.io/devbox-kafka-tls-2           ingress-kafka   kafka-b2.mydomain.com   a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80, 443   23h
ingress.networking.k8s.io/devbox-kafka-tls-bootstrap   ingress-kafka   kafka.mydomain.com      a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80, 443   23h
ingress.networking.k8s.io/ingress-myservicea           ingress-kafka   details.mydomain.com    a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80        24h

kubectl describe ing -n kafka                                                                                   1266ms
Name:             devbox-kafka-tls-0
Labels:           app.kubernetes.io/instance=devbox
                  app.kubernetes.io/managed-by=strimzi-cluster-operator
                  app.kubernetes.io/name=kafka
                  app.kubernetes.io/part-of=strimzi-devbox
                  strimzi.io/cluster=devbox
                  strimzi.io/component-type=kafka
                  strimzi.io/kind=Kafka
                  strimzi.io/name=devbox-kafka
Namespace:        kafka
Address:          a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com
Ingress Class:    ingress-kafka
Default backend:  <default>
TLS:
  SNI routes kafka-b0.mydomain.com
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  kafka-b0.mydomain.com  
                               /   devbox-kafka-tls-0:9094 (10.0.9.68:9094)
Annotations:                   external-dns.alpha.kubernetes.io/hostname: kafka-b0.mydomain.com.
                               external-dns.alpha.kubernetes.io/ttl: 600
                               field.cattle.io/publicEndpoints:
                                 [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-0","ingressName":"kafka:devbox-kafka-tls-0","hostna...
                               ingress.kubernetes.io/ssl-passthrough: true
                               nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                               nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                     From                      Message
  ----    ------  ----                    ----                      -------
  Normal  Sync    5m19s (x1262 over 21h)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    76s (x8 over 3m17s)     nginx-ingress-controller  Scheduled for sync

Name:             devbox-kafka-tls-1
Labels:           app.kubernetes.io/instance=devbox
                  app.kubernetes.io/managed-by=strimzi-cluster-operator
                  app.kubernetes.io/name=kafka
                  app.kubernetes.io/part-of=strimzi-devbox
                  strimzi.io/cluster=devbox
                  strimzi.io/component-type=kafka
                  strimzi.io/kind=Kafka
                  strimzi.io/name=devbox-kafka
Namespace:        kafka
Address:          a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com
Ingress Class:    ingress-kafka
Default backend:  <default>
TLS:
  SNI routes kafka-b1.mydomain.com
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  kafka-b1.mydomain.com  
                               /   devbox-kafka-tls-1:9094 (10.0.14.93:9094)
Annotations:                   external-dns.alpha.kubernetes.io/hostname: kafka-b1.mydomain.com.
                               external-dns.alpha.kubernetes.io/ttl: 600
                               field.cattle.io/publicEndpoints:
                                 [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-1","ingressName":"kafka:devbox-kafka-tls-1","hostna...
                               ingress.kubernetes.io/ssl-passthrough: true
                               nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                               nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                     From                      Message
  ----    ------  ----                    ----                      -------
  Normal  Sync    5m19s (x1262 over 21h)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    76s (x8 over 3m17s)     nginx-ingress-controller  Scheduled for sync

Name:             devbox-kafka-tls-2
Labels:           app.kubernetes.io/instance=devbox
                  app.kubernetes.io/managed-by=strimzi-cluster-operator
                  app.kubernetes.io/name=kafka
                  app.kubernetes.io/part-of=strimzi-devbox
                  strimzi.io/cluster=devbox
                  strimzi.io/component-type=kafka
                  strimzi.io/kind=Kafka
                  strimzi.io/name=devbox-kafka
Namespace:        kafka
Address:          a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com
Ingress Class:    ingress-kafka
Default backend:  <default>
TLS:
  SNI routes kafka-b2.mydomain.com
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  kafka-b2.mydomain.com  
                               /   devbox-kafka-tls-2:9094 (10.0.12.200:9094)
Annotations:                   external-dns.alpha.kubernetes.io/hostname: kafka-b2.mydomain.com.
                               external-dns.alpha.kubernetes.io/ttl: 600
                               field.cattle.io/publicEndpoints:
                                 [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-2","ingressName":"kafka:devbox-kafka-tls-2","hostna...
                               ingress.kubernetes.io/ssl-passthrough: true
                               nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                               nginx.ingress.kubernetes.io/force-ssl-redirect: true
                               nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                     From                      Message
  ----    ------  ----                    ----                      -------
  Normal  Sync    5m20s (x1262 over 21h)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    77s (x8 over 3m18s)     nginx-ingress-controller  Scheduled for sync

Name:             devbox-kafka-tls-bootstrap
Labels:           app.kubernetes.io/instance=devbox
                  app.kubernetes.io/managed-by=strimzi-cluster-operator
                  app.kubernetes.io/name=kafka
                  app.kubernetes.io/part-of=strimzi-devbox
                  strimzi.io/cluster=devbox
                  strimzi.io/component-type=kafka
                  strimzi.io/kind=Kafka
                  strimzi.io/name=devbox-kafka
Namespace:        kafka
Address:          a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com
Ingress Class:    ingress-kafka
Default backend:  <default>
TLS:
  SNI routes kafka.mydomain.com
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  kafka.mydomain.com  
                            /   devbox-kafka-tls-bootstrap:9094 (10.0.12.200:9094,10.0.14.93:9094,10.0.9.68:9094)
Annotations:                external-dns.alpha.kubernetes.io/hostname: kafka.mydomain.com.
                            external-dns.alpha.kubernetes.io/ttl: 600
                            field.cattle.io/publicEndpoints:
                              [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-bootstrap","ingressName":"kafka:devbox-kafka-tls-bo...
                            ingress.kubernetes.io/ssl-passthrough: true
                            nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                            nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                     From                      Message
  ----    ------  ----                    ----                      -------
  Normal  Sync    5m20s (x1262 over 21h)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    77s (x8 over 3m18s)     nginx-ingress-controller  Scheduled for sync

Name:             ingress-myservicea
Labels:           <none>
Namespace:        kafka
Address:          a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com
Ingress Class:    ingress-kafka
Default backend:  <default>
Rules:
  Host                        Path  Backends
  ----                        ----  --------
  details.mydomain.com  
                              /   details:443 (10.0.2.151:443)
Annotations:                  external-dns.alpha.kubernetes.io/hostname: details.mydomain.com.
                              external-dns.alpha.kubernetes.io/ttl: 600
                              field.cattle.io/publicEndpoints:
                                [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"kafka:details","ingressName":"kafka:ingress-myservicea","hostname":"details....
                              nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    4m57s (x2 over 21h)    nginx-ingress-controller  Scheduled for sync
  Normal  Sync    2m38s (x4 over 3m19s)  nginx-ingress-controller  Scheduled for sync

nginx ingress controller pod logs

  NGINX Ingress controller
  Release:       v1.6.4
  Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0222 15:09:57.228198       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0222 15:09:57.228332       7 main.go:209] "Creating API client" host="https://172.20.0.1:443"
I0222 15:09:57.244126       7 main.go:253] "Running in Kubernetes cluster" major="1" minor="23+" git="v1.23.14-eks-ffeb93d" state="clean" commit="96e7d52c98a32f2b296ca7f19dc9346cf79915ba" platform="linux/amd64"
I0222 15:09:57.476396       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0222 15:09:57.499733       7 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0222 15:09:57.518278       7 nginx.go:261] "Starting NGINX Ingress controller"

I0222 15:09:58.629764       7 store.go:433] "Found valid IngressClass" ingress="kafka/devbox-kafka-tls-1" ingressclass="ingress-kafka"
I0222 15:09:58.630441       7 store.go:433] "Found valid IngressClass" ingress="kafka/ingress-myservicea" ingressclass="ingress-kafka"

I0222 15:09:58.724469       7 nginx.go:755] "Starting TLS proxy for SSL Passthrough"
I0222 15:09:58.724509       7 leaderelection.go:248] attempting to acquire leader lease kafka/kafka-ingress-ingress-nginx-leader...
I0222 15:09:58.724573       7 nginx.go:304] "Starting NGINX process"

longwuyuan commented 1 year ago

Curl is going to 10.0.0.180 and ingress shows external-ip as a AWS FQDN so not surprised

On Wed, 22 Feb, 2023, 8:58 pm iyenigul, @.***> wrote:

@longwuyuan https://github.com/longwuyuan still does not work for AWS NLB There are so many ingress for kafka . here i am mainly testing SSL passthrough over details.mydomain.com ingress and pod. so please dont worry about kafka ones and let's focus on details.mydomain.com only in the following output.

curl output for ingress url:

$ curl -v https://details.mydomain.com/

Trying 10.0.0.180:443...

Connected to details.mydomain.com (10.0.0.180) port 443 (#0)

ALPN: offers h2

ALPN: offers http/1.1

CAfile: /etc/ssl/cert.pem

CApath: none

(304) (OUT), TLS handshake, Client hello (1):

(304) (IN), TLS handshake, Server hello (2):

TLSv1.2 (IN), TLS handshake, Certificate (11):

TLSv1.2 (IN), TLS handshake, Server key exchange (12):

TLSv1.2 (IN), TLS handshake, Server finished (14):

TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

TLSv1.2 (OUT), TLS handshake, Finished (20):

TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

TLSv1.2 (IN), TLS handshake, Finished (20):

SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

ALPN: server did not agree on a protocol. Uses default.

Server certificate:

subject: CN=*.mydomain.com

start date: Aug 14 00:00:00 2022 GMT

expire date: Sep 12 23:59:59 2023 GMT

subjectAltName: host "details.mydomain.com" matched cert's "*.mydomain.com"

issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon

SSL certificate verify ok.

GET / HTTP/1.1 Host: details.mydomain.com User-Agent: curl/7.86.0 Accept: /

Mark bundle as not supporting multiuse < HTTP/1.1 400 Bad Request < Date: Wed, 22 Feb 2023 15:08:30 GMT < Content-Type: text/html < Content-Length: 248 < Connection: close < 400 The plain HTTP request was sent to HTTPS port
400 Bad Request
The plain HTTP request was sent to HTTPS port
nginx

Closing connection 0

TLSv1.2 (OUT), TLS alert, close notify (256):

log for related to curl command:

127.0.0.1 - - [22/Feb/2023:15:18:27 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.86.0" 90 0.000 [] [] - - - - 2c04d78844a6875b835728360db2379e

As you see above, I see the SSL details from AWS certificate, not from the pod which is self signed.

curl test from k8s inside from a pod to details pod

bash-5.1$ curl -v https://10.0.2.151 -k

Trying 10.0.2.151:443...

Connected to 10.0.2.151 (10.0.2.151) port 443 (#0)

ALPN, offering h2

ALPN, offering http/1.1

successfully set certificate verify locations:

CAfile: /etc/ssl/certs/ca-certificates.crt

CApath: none

TLSv1.3 (OUT), TLS handshake, Client hello (1):

TLSv1.3 (IN), TLS handshake, Server hello (2):

TLSv1.2 (IN), TLS handshake, Certificate (11):

TLSv1.2 (IN), TLS handshake, Server key exchange (12):

TLSv1.2 (IN), TLS handshake, Server finished (14):

TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

TLSv1.2 (OUT), TLS handshake, Finished (20):

TLSv1.2 (IN), TLS handshake, Finished (20):

SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

ALPN, server accepted to use h2

Server certificate:

subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED

start date: Dec 5 10:37:17 2015 GMT

expire date: Nov 30 10:37:17 2035 GMT

issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED

SSL certificate verify result: self signed certificate (18), continuing anyway.

Using HTTP2, server supports multiplexing

Connection state changed (HTTP/2 confirmed)

Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

Using Stream ID: 1 (easy handle 0x559e45fa26c0)

GET / HTTP/2 Host: 10.0.2.151 user-agent: curl/7.78.0 accept: /

Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 < server: nginx < date: Wed, 22 Feb 2023 15:21:00 GMT < content-type: text/html < content-length: 5 < last-modified: Tue, 21 Feb 2023 16:48:27 GMT < etag: "63f4f5db-5" < accept-ranges: bytes < test

Connection #0 to host 10.0.2.151 left intact

my ingress.yml

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myservicea namespace: kafka annotations: external-dns.alpha.kubernetes.io/hostname: details.mydomain.com. external-dns.alpha.kubernetes.io/ttl: "600" nginx.ingress.kubernetes.io/ssl-passthrough: "true" spec: ingressClassName: ingress-kafka rules:

host: details.mydomain.com http: paths:

path: / pathType: Prefix backend: service: name: details port: number: 443

kubectl svc and ing details in kafka namespace

kubectl get svc,ing -o wide -n kafka 1866ms

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/details NodePort 172.20.80.93 443:32006/TCP 24h app=details service/devbox-cruise-control ClusterIP 172.20.106.66 9090/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-cruise-control service/devbox-kafka-bootstrap http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-cruise-controlservice/devbox-kafka-bootstrap ClusterIP 172.20.53.26 9091/TCP,9092/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-brokers http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-brokers ClusterIP None 9090/TCP,9091/TCP,9092/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-0 http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-0 ClusterIP 172.20.110.50 9094/TCP 23h statefulset.kubernetes.io/pod-name=devbox-kafka-0,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-1 http://statefulset.kubernetes.io/pod-name=devbox-kafka-0,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-1 ClusterIP 172.20.197.234 9094/TCP 23h statefulset.kubernetes.io/pod-name=devbox-kafka-1,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-2 http://statefulset.kubernetes.io/pod-name=devbox-kafka-1,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-2 ClusterIP 172.20.243.234 9094/TCP 23h statefulset.kubernetes.io/pod-name=devbox-kafka-2,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-bootstrap http://statefulset.kubernetes.io/pod-name=devbox-kafka-2,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-bootstrap ClusterIP 172.20.10.227 9094/TCP 23h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-zookeeper-client http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-zookeeper-client ClusterIP 172.20.3.146 2181/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeper service/devbox-zookeeper-nodes http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeperservice/devbox-zookeeper-nodes ClusterIP None 2181/TCP,2888/TCP,3888/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeper service/kafka-ingress-ingress-nginx-controller http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeperservice/kafka-ingress-ingress-nginx-controller LoadBalancer 172.20.38.32 a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80:32352/TCP,443:31567/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx service/kafka-ingress-ingress-nginx-controller-admission http://app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginxservice/kafka-ingress-ingress-nginx-controller-admission ClusterIP 172.20.5.5 443/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx

NAME CLASS HOSTS ADDRESS PORTS AGEingress.networking.k8s.io/devbox-kafka-tls-0 ingress-kafka kafka-b0.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/devbox-kafka-tls-1 ingress-kafka kafka-b1.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/devbox-kafka-tls-2 ingress-kafka kafka-b2.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/devbox-kafka-tls-bootstrap ingress-kafka kafka.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/ingress-myservicea ingress-kafka details.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80 24h

kubectl describe ing -n kafka 1266ms Name: devbox-kafka-tls-0 Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka-b0.mydomain.com Rules: Host Path Backends

kafka-b0.mydomain.com / devbox-kafka-tls-0:9094 (10.0.9.68:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka-b0.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-0","ingressName":"kafka:devbox-kafka-tls-0","hostna... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m19s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 76s (x8 over 3m17s) nginx-ingress-controller Scheduled for sync

Name: devbox-kafka-tls-1 Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka-b1.mydomain.com Rules: Host Path Backends

kafka-b1.mydomain.com / devbox-kafka-tls-1:9094 (10.0.14.93:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka-b1.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-1","ingressName":"kafka:devbox-kafka-tls-1","hostna... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m19s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 76s (x8 over 3m17s) nginx-ingress-controller Scheduled for sync

Name: devbox-kafka-tls-2 Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka-b2.mydomain.com Rules: Host Path Backends

kafka-b2.mydomain.com / devbox-kafka-tls-2:9094 (10.0.12.200:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka-b2.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-2","ingressName":"kafka:devbox-kafka-tls-2","hostna... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/force-ssl-redirect: true nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m20s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 77s (x8 over 3m18s) nginx-ingress-controller Scheduled for sync

Name: devbox-kafka-tls-bootstrap Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka.mydomain.com Rules: Host Path Backends

kafka.mydomain.com / devbox-kafka-tls-bootstrap:9094 (10.0.12.200:9094,10.0.14.93:9094,10.0.9.68:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-bootstrap","ingressName":"kafka:devbox-kafka-tls-bo... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m20s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 77s (x8 over 3m18s) nginx-ingress-controller Scheduled for sync

Name: ingress-myservicea Labels: Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: Rules: Host Path Backends

details.mydomain.com / details:443 (10.0.2.151:443) Annotations: external-dns.alpha.kubernetes.io/hostname: details.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"kafka:details","ingressName":"kafka:ingress-myservicea","hostname":"details.... nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 4m57s (x2 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 2m38s (x4 over 3m19s) nginx-ingress-controller Scheduled for sync

nginx ingress controller pod logs

NGINX Ingress controller Release: v1.6.4 Build: 69e8833858fb6bda12a44990f1d5eaa7b13f4b75 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6

W0222 15:09:57.228198 7 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0222 15:09:57.228332 7 main.go:209] "Creating API client" host="https://172.20.0.1:443" I0222 15:09:57.244126 7 main.go:253] "Running in Kubernetes cluster" major="1" minor="23+" git="v1.23.14-eks-ffeb93d" state="clean" commit="96e7d52c98a32f2b296ca7f19dc9346cf79915ba" platform="linux/amd64" I0222 15:09:57.476396 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem" I0222 15:09:57.499733 7 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key" I0222 15:09:57.518278 7 nginx.go:261] "Starting NGINX Ingress controller"

I0222 15:09:58.629764 7 store.go:433] "Found valid IngressClass" ingress="kafka/devbox-kafka-tls-1" ingressclass="ingress-kafka" I0222 15:09:58.630441 7 store.go:433] "Found valid IngressClass" ingress="kafka/ingress-myservicea" ingressclass="ingress-kafka"

I0222 15:09:58.724469 7 nginx.go:755] "Starting TLS proxy for SSL Passthrough" I0222 15:09:58.724509 7 leaderelection.go:248] attempting to acquire leader lease kafka/kafka-ingress-ingress-nginx-leader... I0222 15:09:58.724573 7 nginx.go:304] "Starting NGINX process"

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/9658#issuecomment-1440256024, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWRJ7PQYZQKI7DH2YW3WYYWAVANCNFSM6AAAAAAVDMLQZ4 . You are receiving this because you were mentioned.Message ID: @.***>

ismailyenigul commented 1 year ago

sorry? This is internal AWS load balancer.

$ host details.mydomain.com           
details.mydomain.com     has address 10.0.2.129
details.mydomain.com    has address 10.0.4.28
details.mydomain.com   has address 10.0.0.180

longwuyuan commented 1 year ago

But you got valid cert so ingress routing worked. The error seen is http request sent to https listener.

On Wed, 22 Feb, 2023, 9:08 pm Yuan, @.***> wrote:

Curl is going to 10.0.0.180 and ingress shows external-ip as a AWS FQDN so not surprised

On Wed, 22 Feb, 2023, 8:58 pm iyenigul, @.***> wrote:

@longwuyuan https://github.com/longwuyuan still does not work for AWS NLB There are so many ingress for kafka . here i am mainly testing SSL passthrough over details.mydomain.com ingress and pod. so please dont worry about kafka ones and let's focus on details.mydomain.com only in the following output.

curl output for ingress url:

$ curl -v https://details.mydomain.com/

Trying 10.0.0.180:443...

Connected to details.mydomain.com (10.0.0.180) port 443 (#0)

ALPN: offers h2

ALPN: offers http/1.1

CAfile: /etc/ssl/cert.pem

CApath: none

(304) (OUT), TLS handshake, Client hello (1):

(304) (IN), TLS handshake, Server hello (2):

TLSv1.2 (IN), TLS handshake, Certificate (11):

TLSv1.2 (IN), TLS handshake, Server key exchange (12):

TLSv1.2 (IN), TLS handshake, Server finished (14):

TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

TLSv1.2 (OUT), TLS handshake, Finished (20):

TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

TLSv1.2 (IN), TLS handshake, Finished (20):

SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

ALPN: server did not agree on a protocol. Uses default.

Server certificate:

subject: CN=*.mydomain.com

start date: Aug 14 00:00:00 2022 GMT

expire date: Sep 12 23:59:59 2023 GMT

subjectAltName: host "details.mydomain.com" matched cert's "*.mydomain.com"

issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon

SSL certificate verify ok.

GET / HTTP/1.1 Host: details.mydomain.com User-Agent: curl/7.86.0 Accept: /

Mark bundle as not supporting multiuse < HTTP/1.1 400 Bad Request < Date: Wed, 22 Feb 2023 15:08:30 GMT < Content-Type: text/html < Content-Length: 248 < Connection: close < 400 The plain HTTP request was sent to HTTPS port
400 Bad Request
The plain HTTP request was sent to HTTPS port
nginx

Closing connection 0

TLSv1.2 (OUT), TLS alert, close notify (256):

log for related to curl command:

127.0.0.1 - - [22/Feb/2023:15:18:27 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.86.0" 90 0.000 [] [] - - - - 2c04d78844a6875b835728360db2379e

As you see above, I see the SSL details from AWS certificate, not from the pod which is self signed.

curl test from k8s inside from a pod to details pod

bash-5.1$ curl -v https://10.0.2.151 -k

Trying 10.0.2.151:443...

Connected to 10.0.2.151 (10.0.2.151) port 443 (#0)

ALPN, offering h2

ALPN, offering http/1.1

successfully set certificate verify locations:

CAfile: /etc/ssl/certs/ca-certificates.crt

CApath: none

TLSv1.3 (OUT), TLS handshake, Client hello (1):

TLSv1.3 (IN), TLS handshake, Server hello (2):

TLSv1.2 (IN), TLS handshake, Certificate (11):

TLSv1.2 (IN), TLS handshake, Server key exchange (12):

TLSv1.2 (IN), TLS handshake, Server finished (14):

TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

TLSv1.2 (OUT), TLS handshake, Finished (20):

TLSv1.2 (IN), TLS handshake, Finished (20):

SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

ALPN, server accepted to use h2

Server certificate:

subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED

start date: Dec 5 10:37:17 2015 GMT

expire date: Nov 30 10:37:17 2035 GMT

issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED

SSL certificate verify result: self signed certificate (18), continuing anyway.

Using HTTP2, server supports multiplexing

Connection state changed (HTTP/2 confirmed)

Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

Using Stream ID: 1 (easy handle 0x559e45fa26c0)

GET / HTTP/2 Host: 10.0.2.151 user-agent: curl/7.78.0 accept: /

Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 < server: nginx < date: Wed, 22 Feb 2023 15:21:00 GMT < content-type: text/html < content-length: 5 < last-modified: Tue, 21 Feb 2023 16:48:27 GMT < etag: "63f4f5db-5" < accept-ranges: bytes < test

Connection #0 to host 10.0.2.151 left intact

my ingress.yml

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myservicea namespace: kafka annotations: external-dns.alpha.kubernetes.io/hostname: details.mydomain.com. external-dns.alpha.kubernetes.io/ttl: "600" nginx.ingress.kubernetes.io/ssl-passthrough: "true" spec: ingressClassName: ingress-kafka rules:

host: details.mydomain.com http: paths:

path: / pathType: Prefix backend: service: name: details port: number: 443

kubectl svc and ing details in kafka namespace

kubectl get svc,ing -o wide -n kafka 1866ms

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/details NodePort 172.20.80.93 443:32006/TCP 24h app=details service/devbox-cruise-control ClusterIP 172.20.106.66 9090/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-cruise-control service/devbox-kafka-bootstrap http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-cruise-controlservice/devbox-kafka-bootstrap ClusterIP 172.20.53.26 9091/TCP,9092/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-brokers http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-brokers ClusterIP None 9090/TCP,9091/TCP,9092/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-0 http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-0 ClusterIP 172.20.110.50 9094/TCP 23h statefulset.kubernetes.io/pod-name=devbox-kafka-0,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-1 http://statefulset.kubernetes.io/pod-name=devbox-kafka-0,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-1 ClusterIP 172.20.197.234 9094/TCP 23h statefulset.kubernetes.io/pod-name=devbox-kafka-1,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-2 http://statefulset.kubernetes.io/pod-name=devbox-kafka-1,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-2 ClusterIP 172.20.243.234 9094/TCP 23h statefulset.kubernetes.io/pod-name=devbox-kafka-2,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-kafka-tls-bootstrap http://statefulset.kubernetes.io/pod-name=devbox-kafka-2,strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-kafka-tls-bootstrap ClusterIP 172.20.10.227 9094/TCP 23h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafka service/devbox-zookeeper-client http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-kafkaservice/devbox-zookeeper-client ClusterIP 172.20.3.146 2181/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeper service/devbox-zookeeper-nodes http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeperservice/devbox-zookeeper-nodes ClusterIP None 2181/TCP,2888/TCP,3888/TCP 24h strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeper service/kafka-ingress-ingress-nginx-controller http://strimzi.io/cluster=devbox,strimzi.io/kind=Kafka,strimzi.io/name=devbox-zookeeperservice/kafka-ingress-ingress-nginx-controller LoadBalancer 172.20.38.32 a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80:32352/TCP,443:31567/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx service/kafka-ingress-ingress-nginx-controller-admission http://app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginxservice/kafka-ingress-ingress-nginx-controller-admission ClusterIP 172.20.5.5 443/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx

NAME CLASS HOSTS ADDRESS PORTS AGEingress.networking.k8s.io/devbox-kafka-tls-0 ingress-kafka kafka-b0.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/devbox-kafka-tls-1 ingress-kafka kafka-b1.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/devbox-kafka-tls-2 ingress-kafka kafka-b2.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/devbox-kafka-tls-bootstrap ingress-kafka kafka.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80, 443 23hingress.networking.k8s.io/ingress-myservicea ingress-kafka details.mydomain.com a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com 80 24h

kubectl describe ing -n kafka 1266ms Name: devbox-kafka-tls-0 Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka-b0.mydomain.com Rules: Host Path Backends

kafka-b0.mydomain.com / devbox-kafka-tls-0:9094 (10.0.9.68:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka-b0.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-0","ingressName":"kafka:devbox-kafka-tls-0","hostna... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m19s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 76s (x8 over 3m17s) nginx-ingress-controller Scheduled for sync

Name: devbox-kafka-tls-1 Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka-b1.mydomain.com Rules: Host Path Backends

kafka-b1.mydomain.com / devbox-kafka-tls-1:9094 (10.0.14.93:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka-b1.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-1","ingressName":"kafka:devbox-kafka-tls-1","hostna... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m19s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 76s (x8 over 3m17s) nginx-ingress-controller Scheduled for sync

Name: devbox-kafka-tls-2 Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka-b2.mydomain.com Rules: Host Path Backends

kafka-b2.mydomain.com / devbox-kafka-tls-2:9094 (10.0.12.200:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka-b2.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-2","ingressName":"kafka:devbox-kafka-tls-2","hostna... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/force-ssl-redirect: true nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m20s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 77s (x8 over 3m18s) nginx-ingress-controller Scheduled for sync

Name: devbox-kafka-tls-bootstrap Labels: app.kubernetes.io/instance=devbox app.kubernetes.io/managed-by=strimzi-cluster-operator app.kubernetes.io/name=kafka app.kubernetes.io/part-of=strimzi-devbox strimzi.io/cluster=devbox strimzi.io/component-type=kafka strimzi.io/kind=Kafka strimzi.io/name=devbox-kafka Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: TLS: SNI routes kafka.mydomain.com Rules: Host Path Backends

kafka.mydomain.com / devbox-kafka-tls-bootstrap:9094 (10.0.12.200:9094,10.0.14.93:9094,10.0.9.68:9094) Annotations: external-dns.alpha.kubernetes.io/hostname: kafka.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:devbox-kafka-tls-bootstrap","ingressName":"kafka:devbox-kafka-tls-bo... ingress.kubernetes.io/ssl-passthrough: true nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 5m20s (x1262 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 77s (x8 over 3m18s) nginx-ingress-controller Scheduled for sync

Name: ingress-myservicea Labels: Namespace: kafka Address: a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com Ingress Class: ingress-kafka Default backend: Rules: Host Path Backends

details.mydomain.com / details:443 (10.0.2.151:443) Annotations: external-dns.alpha.kubernetes.io/hostname: details.mydomain.com. external-dns.alpha.kubernetes.io/ttl: 600 field.cattle.io/publicEndpoints: [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"kafka:details","ingressName":"kafka:ingress-myservicea","hostname":"details.... nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message

Normal Sync 4m57s (x2 over 21h) nginx-ingress-controller Scheduled for sync Normal Sync 2m38s (x4 over 3m19s) nginx-ingress-controller Scheduled for sync

nginx ingress controller pod logs

NGINX Ingress controller Release: v1.6.4 Build: 69e8833858fb6bda12a44990f1d5eaa7b13f4b75 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6

W0222 15:09:57.228198 7 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0222 15:09:57.228332 7 main.go:209] "Creating API client" host="https://172.20.0.1:443" I0222 15:09:57.244126 7 main.go:253] "Running in Kubernetes cluster" major="1" minor="23+" git="v1.23.14-eks-ffeb93d" state="clean" commit="96e7d52c98a32f2b296ca7f19dc9346cf79915ba" platform="linux/amd64" I0222 15:09:57.476396 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem" I0222 15:09:57.499733 7 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key" I0222 15:09:57.518278 7 nginx.go:261] "Starting NGINX Ingress controller"

I0222 15:09:58.629764 7 store.go:433] "Found valid IngressClass" ingress="kafka/devbox-kafka-tls-1" ingressclass="ingress-kafka" I0222 15:09:58.630441 7 store.go:433] "Found valid IngressClass" ingress="kafka/ingress-myservicea" ingressclass="ingress-kafka"

I0222 15:09:58.724469 7 nginx.go:755] "Starting TLS proxy for SSL Passthrough" I0222 15:09:58.724509 7 leaderelection.go:248] attempting to acquire leader lease kafka/kafka-ingress-ingress-nginx-leader... I0222 15:09:58.724573 7 nginx.go:304] "Starting NGINX process"

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/9658#issuecomment-1440256024, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWRJ7PQYZQKI7DH2YW3WYYWAVANCNFSM6AAAAAAVDMLQZ4 . You are receiving this because you were mentioned.Message ID: @.***>

ismailyenigul commented 1 year ago

yes! it means somehow nginx does not send https request to the pod.

longwuyuan commented 1 year ago

@ismailyenigul The problem to be solved here looks like you needing support on how to configure an ingress. For that I think you should chat on the Kubernetes slack kubernetes.slack.com. Register at slak.k8s.io if needed. There are not many resources here on github to provide support in the way you seek.

Sometimes if the related information is clear then its possible to resolve issues. But in this case, you have posted a curl and the output of curl shows a certificate but you say that is a problem. It seems you wanted to see a different certificate that is self-signed. That kind of troubleshooting can be done on a one-one-one session by walking you through the configuration and the results of tests etc. Its better done on chat.

If there is a problem with the functional aspects of the controller then its better to discuss here in github so the code can be fixed. I have tested and there is no problem discovered so far with the ssl-passthrough annotation.

ismailyenigul commented 1 year ago

@longwuyuan I thought that in nginx SSL passtrought case, client will see certificate from pod instead of ALB aWS cert. Maybe I am wrong but it is clear that nginx-ingress try to connect pod in HTTP scheme not HTTPS. So please focus on why nginx-ingress does not send HTTPS request when I set nginx.ingress.kubernetes.io/ssl-passthrough: true instead of telling me to get help for ingress. I am working on ingress for 5 years and did so many deployments with ALB ingress controller and nginx-ingress controller. but all with HTTP backend. This is the first time I try to terminate SSL on pod instead of LB. So please tell me why nginx-ingress send HTTP to pod with nginx.ingress.kubernetes.io/ssl-passthrough: true ? Please test with AWS env instead of minikube which is not used in production. Also I am pretty sure that AWS NLB send https requests to nginx-ingress HTTPS nodeport. It is clear that nginx receives HTTPS request.

longwuyuan commented 1 year ago

Hi @ismailyenigul , please understand, its required to balance between support and other activities and I requested to you to continue talking on Kubernetes slack for support as there are more people who can help solve your problem.

If you think the controller code is broken, then reopen the ticket after you have pasted proof and data that a developer can look at.

longwuyuan commented 1 year ago

One way to make progress here is do these steps ;

Use these 3 annotations only

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"

Post the following information
- Complete curl command with -v and its reponse
- kubectl describe ing $ingressname
- kubectl describe pod $controllerpodname
- kubectl logs $controllerpodname
- kubectl logs $applicationpodname

ismailyenigul commented 1 year ago

Here is the screenshot of AWS NLB. As you NLB forwards 443 request to HTTPS port of service/kafka-ingress-ingress-nginx-controller (port 31567)

service/kafka-ingress-ingress-nginx-controller             LoadBalancer   172.20.38.32     a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.myregion.amazonaws.com   80:32352/TCP,443:31567/TCP   21h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=kafka-ingress,app.kubernetes.io/name=ingress-nginx

Name:             ingress-myservicea
Labels:           <none>
Namespace:        kafka
Address:          a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.us-east-1.amazonaws.com
Ingress Class:    ingress-kafka
Default backend:  <default>
Rules:
  Host                        Path  Backends
  ----                        ----  --------
  details.devbox-apps.nd.com  
                              /   details:443 (10.0.0.171:443)
Annotations:                  external-dns.alpha.kubernetes.io/hostname: details.devbox-apps.nd.com.
                              external-dns.alpha.kubernetes.io/ttl: 600
                              field.cattle.io/publicEndpoints:
                                [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"kafka:details","ingressName":"kafka:ingress-myservicea","hostname":"details....
                              nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                              nginx.ingress.kubernetes.io/ssl-passthrough: true
                              nginx.ingress.kubernetes.io/ssl-redirect: true
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    4m49s (x5 over 164m)  nginx-ingress-controller  Scheduled for sync

curl output:

$ curl -v https://details.mydomain.com/ismail
*   Trying 10.0.4.28:443...
* Connected to details.mydomain.com (10.0.4.28) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Aug 14 00:00:00 2022 GMT
*  expire date: Sep 12 23:59:59 2023 GMT
*  subjectAltName: host "details.mydomain.com" matched cert's "*.mydomain.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: details.mydomain.com
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Thu, 23 Feb 2023 05:53:36 GMT
< Content-Type: text/html
< Content-Length: 248
< Connection: close
< 
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0
* TLSv1.2 (IN), TLS alert, close notify (256):
* TLSv1.2 (OUT), TLS alert, close notify (256):

nginx ingress controller logs:

127.0.0.1 - - [23/Feb/2023:05:53:36 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.86.0" 90 0.000 [] [] - - - - aea4d64b3c4e3ea653a7afa60d0721bb

I enabled debug in nginx-controller with --v=5

2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 28#28: *1148 free: 00007F9C44EE88D0, unused: 136
2023-02-23T08:56:30+03:00 I0223 05:56:30.336277       6 tcp.go:84] "passing to" hostport="127.0.0.1:442"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: accept on 0.0.0.0:442, ready: 1
2023-02-23T08:56:30+03:00 I0223 05:56:30.336401       6 tcp.go:103] "Writing Proxy Protocol" header=<
2023-02-23T08:56:30+03:00   PROXY TCP4 10.0.0.67 10.0.2.94 17259 443
2023-02-23T08:56:30+03:00  >
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: posix_memalign: 00007F9C44EE8D50:512 @16
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 accept: 127.0.0.1:47264 fd:3
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 event timer add: 3: 60000:10251289
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 reusable connection: 1
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 epoll add event: fd:3 op:1 ev:80002001
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: accept() not ready (11: Resource temporarily unavailable)
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http check ssl handshake
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http recv(): 108
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 PROXY protocol src: 10.0.0.67 17259, dst: 10.0.2.94 443
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 recv: eof:0, avail:-1
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 recv: fd:3 42 of 42
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 recv: avail:96
**2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 plain http**
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 plain http
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http wait request handler
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 malloc: 00007F9C44E83240:1024
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 recv: eof:0, avail:96
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 recv: fd:3 96 of 1024
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 recv: avail:0
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 reusable connection: 0
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 posix_memalign: 00007F9C469FC120:4096 @16
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http process request line
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http request line: "GET /ismail HTTP/1.1"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http uri: "/ismail"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http args: ""
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http exten: ""
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 posix_memalign: 00007F9C469FD360:4096 @16
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http process request header line
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http header: "Host: details.mydomain.com"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http header: "User-Agent: curl/7.86.0"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http header: "Accept: */*"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http header done
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [info] 30#30: *1149 client sent plain HTTP request to HTTPS port while reading client request headers, client: 127.0.0.1, server: details.mydomain.com, request: "GET /ismail HTTP/1.1", host: "details.mydomain.com"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http finalize request: 497, "/ismail?" a:1, c:1
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 event timer del: 3: 10251289
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http special response: 497, "/ismail?"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 http set discard body
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 lua header filter for user lua code, uri "/ismail"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 lua capture header filter, uri "/ismail"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 headers more header filter, uri "/ismail"
2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 HTTP/1.1 400 Bad Request

@longwuyuan

and the following tell us that SSL proxy is in use

2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: accept on 0.0.0.0:442, ready: 1
2023-02-23T08:56:30+03:00 I0223 05:56:30.336401       6 tcp.go:103] "Writing Proxy Protocol" header=<
2023-02-23T08:56:30+03:00   PROXY TCP4 10.0.0.67 10.0.2.94 17259 443

but it connects to backend pod port in HTTP plain instead of HTTPS.

see the following line. the protocol is HTTP

2023-02-23T08:56:30+03:00 2023/02/23 05:56:30 [debug] 30#30: *1149 plain http

somehow, nginx-ingress controller can't get backend scheme. nginx.ingress.kubernetes.io/backend-protocol: HTTPS might not be considered by nginx ingress controller.

I will continue debugging nginx settings for scheme

longwuyuan commented 1 year ago

how was the controller installed ?

ismailyenigul commented 1 year ago

with following helm values

ingress-nginx:
  controller:
    extraArgs: {"enable-ssl-passthrough": ""}
    containerPort:
      https: 443
    ingressClass: ingress-kafka
    ingressClassResource:
      name: ingress-kafka
      enabled: true
      default: false
      controllerValue: "k8s.io/ingress-kafka"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-internal: true
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "100"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:region:id:certificate/certid

longwuyuan commented 1 year ago

I can't test this. Maybe someone else will test and comment
If you wanted to help out, you can easily provide the information that someone can analyze and comment on. For example ;
- Do a tcpdump on the backend pod and check if https arrives or not
- Install the controller as per documented recommended process described here https://kubernetes.github.io/ingress-nginx/deploy/#aws
- I don't know anything about your backend pod. You can give data here about curl from controller pod to backend pod and show what is the response
- So much other detailed informaiton is possible to be provided so someone can analyze

I am not sure if I can help anymore in this situation. Hope other people see and understand what the problem is and are able to resolve this problem for you

ismailyenigul commented 1 year ago

I installed tcpdump on pod and but when I do curl for the url, I dont see any traffic coming to pod 80 or 443. It seems traffic never reaches to pod.

root@details-v1-9f54f4fbb-pknx8:/# netstat -na |grep LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN     
root@details-v1-9f54f4fbb-pknx8:/# tcpdump -X port 80

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@details-v1-9f54f4fbb-pknx8:/# tcpdump -X port 443

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

here is full debug for my ingress annotation and parsing.

I0223 07:32:09.671793       7 main.go:100] "successfully validated configuration, accepting" ingress="kafka/ingress-myservicea"
2023/02/23 07:32:09 [debug] 30#30: *235 http cleanup add: 00007F8E5B012F70
I0223 07:32:09.679754       7 store.go:843] updating annotations information for ingress kafka/ingress-myservicea
I0223 07:32:09.679785       7 annotations.go:187] "Parsing Ingress annotation" name="Redirect" ingress="kafka/ingress-myservicea" value=<nil>
I0223 07:32:09.679800       7 annotations.go:187] "Parsing Ingress annotation" name="SSLCipher" ingress="kafka/ingress-myservicea" value=&{SSLCiphers: SSLPreferServerCiphers:}
I0223 07:32:09.679827       7 annotations.go:187] "Parsing Ingress annotation" name="Whitelist" ingress="kafka/ingress-myservicea" value=&{CIDR:[]}
I0223 07:32:09.679840       7 annotations.go:187] "Parsing Ingress annotation" name="Denylist" ingress="kafka/ingress-myservicea" value=&{CIDR:[]}
I0223 07:32:09.679855       7 annotations.go:187] "Parsing Ingress annotation" name="BackendProtocol" ingress="kafka/ingress-myservicea" value="HTTPS"
I0223 07:32:09.679886       7 annotations.go:187] "Parsing Ingress annotation" name="Proxy" ingress="kafka/ingress-myservicea" value=&{BodySize:1m ConnectTimeout:5 SendTimeout:60 ReadTimeout:60 BuffersNumber:4 BufferSize:4k CookieDomain:off CookiePath:off NextUpstream:error timeout NextUpstreamTimeout:0 NextUpstreamTries:3 ProxyRedirectFrom:off ProxyRedirectTo:off RequestBuffering:on ProxyBuffering:off ProxyHTTPVersion:1.1 ProxyMaxTempFileSize:1024m}
I0223 07:32:09.679895       7 annotations.go:187] "Parsing Ingress annotation" name="ServerSnippet" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.679905       7 main.go:197] "No default affinity found" ingress="ingress-myservicea"
I0223 07:32:09.679924       7 annotations.go:187] "Parsing Ingress annotation" name="SessionAffinity" ingress="kafka/ingress-myservicea" value=&{Type: Mode: CanaryBehavior: Cookie:{Name: Expires: MaxAge: Path: Domain: ChangeOnFailure:false Secure:false SameSite: ConditionalSameSiteNone:false}}
I0223 07:32:09.679942       7 annotations.go:187] "Parsing Ingress annotation" name="Canary" ingress="kafka/ingress-myservicea" value=&{Enabled:false Weight:0 WeightTotal:100 Header: HeaderValue: HeaderPattern: Cookie:}
I0223 07:32:09.679953       7 annotations.go:187] "Parsing Ingress annotation" name="Satisfy" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.679963       7 annotations.go:187] "Parsing Ingress annotation" name="ExternalAuth" ingress="kafka/ingress-myservicea" value=<nil>
I0223 07:32:09.679975       7 annotations.go:187] "Parsing Ingress annotation" name="Opentracing" ingress="kafka/ingress-myservicea" value=&{Enabled:false Set:false TrustEnabled:false TrustSet:false}
I0223 07:32:09.679996       7 annotations.go:187] "Parsing Ingress annotation" name="ProxySSL" ingress="kafka/ingress-myservicea" value=&{AuthSSLCert:{Secret: CAFileName: CASHA: CRLFileName: CRLSHA: PemFileName:} Ciphers: Protocols: ProxySSLName: Verify: VerifyDepth:0 ProxySSLServerName:}
I0223 07:32:09.680006       7 annotations.go:187] "Parsing Ingress annotation" name="ServiceUpstream" ingress="kafka/ingress-myservicea" value=false
I0223 07:32:09.680014       7 annotations.go:187] "Parsing Ingress annotation" name="StreamSnippet" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.680025       7 annotations.go:187] "Parsing Ingress annotation" name="Connection" ingress="kafka/ingress-myservicea" value=&{Header: Enabled:false}
I0223 07:32:09.680033       7 annotations.go:187] "Parsing Ingress annotation" name="CustomHTTPErrors" ingress="kafka/ingress-myservicea" value=<nil>
I0223 07:32:09.680050       7 annotations.go:187] "Parsing Ingress annotation" name="CertificateAuth" ingress="kafka/ingress-myservicea" value=&{AuthSSLCert:{Secret: CAFileName: CASHA: CRLFileName: CRLSHA: PemFileName:} VerifyClient: ValidationDepth:0 ErrorPage: PassCertToUpstream:false MatchCN: AuthTLSError:}
I0223 07:32:09.680061       7 annotations.go:187] "Parsing Ingress annotation" name="ConfigurationSnippet" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.680079       7 annotations.go:187] "Parsing Ingress annotation" name="CorsConfig" ingress="kafka/ingress-myservicea" value=&{CorsEnabled:false CorsAllowOrigin:[*] CorsAllowMethods:GET, PUT, POST, DELETE, PATCH, OPTIONS CorsAllowHeaders:DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization CorsAllowCredentials:true CorsExposeHeaders: CorsMaxAge:1728000}
I0223 07:32:09.680094       7 annotations.go:187] "Parsing Ingress annotation" name="GlobalRateLimit" ingress="kafka/ingress-myservicea" value=&{Namespace: Limit:0 WindowSize:0 Key: IgnoredCIDRs:[]}
I0223 07:32:09.680103       7 annotations.go:187] "Parsing Ingress annotation" name="Aliases" ingress="kafka/ingress-myservicea" value=[]
I0223 07:32:09.680112       7 annotations.go:187] "Parsing Ingress annotation" name="BasicDigestAuth" ingress="kafka/ingress-myservicea" value=<nil>
I0223 07:32:09.680130       7 annotations.go:187] "Parsing Ingress annotation" name="Rewrite" ingress="kafka/ingress-myservicea" value=&{Target: SSLRedirect:true ForceSSLRedirect:false PreserveTrailingSlash:false AppRoot: UseRegex:false}
I0223 07:32:09.680140       7 annotations.go:187] "Parsing Ingress annotation" name="SSLPassthrough" ingress="kafka/ingress-myservicea" value=true
I0223 07:32:09.680152       7 annotations.go:187] "Parsing Ingress annotation" name="UsePortInRedirects" ingress="kafka/ingress-myservicea" value=false
I0223 07:32:09.680165       7 annotations.go:187] "Parsing Ingress annotation" name="UpstreamHashBy" ingress="kafka/ingress-myservicea" value=&{UpstreamHashBy: UpstreamHashBySubset:false UpstreamHashBySubsetSize:3}
I0223 07:32:09.680173       7 annotations.go:187] "Parsing Ingress annotation" name="LoadBalancing" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.680185       7 annotations.go:187] "Parsing Ingress annotation" name="Logs" ingress="kafka/ingress-myservicea" value=&{Access:true Rewrite:false}
I0223 07:32:09.680193       7 annotations.go:187] "Parsing Ingress annotation" name="ClientBodyBufferSize" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.680217       7 annotations.go:187] "Parsing Ingress annotation" name="RateLimit" ingress="kafka/ingress-myservicea" value=&{Connections:{Name: Limit:0 Burst:0 SharedSize:0} RPS:{Name: Limit:0 Burst:0 SharedSize:0} RPM:{Name: Limit:0 Burst:0 SharedSize:0} LimitRate:0 LimitRateAfter:0 Name: ID: Whitelist:[]}
I0223 07:32:09.680226       7 annotations.go:187] "Parsing Ingress annotation" name="SecureUpstream" ingress="kafka/ingress-myservicea" value=<nil>
I0223 07:32:09.680241       7 annotations.go:187] "Parsing Ingress annotation" name="InfluxDB" ingress="kafka/ingress-myservicea" value=&{InfluxDBEnabled:false InfluxDBMeasurement:default InfluxDBPort:8089 InfluxDBHost:127.0.0.1 InfluxDBServerName:nginx-ingress}
I0223 07:32:09.680257       7 annotations.go:187] "Parsing Ingress annotation" name="ModSecurity" ingress="kafka/ingress-myservicea" value=&{Enable:false EnableSet:false OWASPRules:false TransactionID: Snippet:}
I0223 07:32:09.680272       7 annotations.go:187] "Parsing Ingress annotation" name="Mirror" ingress="kafka/ingress-myservicea" value=&{Source: RequestBody:on Target: Host:}
2023/02/23 07:32:09 [debug] 31#31: *236 http cleanup add: 00007F8E5B012F70
2023/02/23 07:32:09 [debug] 30#30: *235 lua run thread, top:0 c:1
2023/02/23 07:32:09 [debug] 31#31: *236 lua run thread, top:0 c:1
I0223 07:32:09.680665       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kafka", Name:"ingress-myservicea", UID:"a4965298-d1c9-4f70-84f1-4e576d5eb4a2", APIVersion:"networking.k8s.io/v1", ResourceVersion:"204011452", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0223 07:32:09.680710       7 annotations.go:187] "Parsing Ingress annotation" name="FastCGI" ingress="kafka/ingress-myservicea" value={Index: Params:map[]}
I0223 07:32:09.680725       7 annotations.go:187] "Parsing Ingress annotation" name="HTTP2PushPreload" ingress="kafka/ingress-myservicea" value=false
I0223 07:32:09.680734       7 annotations.go:187] "Parsing Ingress annotation" name="UpstreamVhost" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.680743       7 annotations.go:187] "Parsing Ingress annotation" name="XForwardedPrefix" ingress="kafka/ingress-myservicea" value=""
I0223 07:32:09.680752       7 annotations.go:187] "Parsing Ingress annotation" name="DefaultBackend" ingress="kafka/ingress-myservicea" value=<nil>
I0223 07:32:09.680761       7 annotations.go:187] "Parsing Ingress annotation" name="EnableGlobalAuth" ingress="kafka/ingress-myservicea" value=true
2023/02/23 07:32:09 [debug] 31#31: *236 lua resume returned 0
I0223 07:32:09.680891       7 store.go:885] updating references to secrets for ingress kafka/ingress-myservicea

but If I do curl to backend pod and nginx controller pod IP directly, I got 200 response

curl -v https://10.0.0.171 -H 'Host: details.mydomain.com'  -k   # backend pod
*   Trying 10.0.0.171:443...
* Connected to 10.0.0.171 (10.0.0.171) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED
*  start date: Dec  5 10:37:17 2015 GMT
*  expire date: Nov 30 10:37:17 2035 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=CERTIFICATE-FILE-NOT-LOADED
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562fe1427700)
> GET / HTTP/2
> Host: details.mydomain.com
> user-agent: curl/7.78.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Thu, 23 Feb 2023 08:49:03 GMT
< content-type: text/html
< content-length: 6
< last-modified: Thu, 23 Feb 2023 08:43:14 GMT
< etag: "63f72722-6"
< accept-ranges: bytes
< 
index

bash-5.1$ curl -v https://10.0.0.105 -H 'Host: details.mydomain.com'  -k  # nginx ingress pod
*   Trying 10.0.0.105:443...
* Connected to 10.0.0.105 (10.0.0.105) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb 23 07:31:54 2023 GMT
*  expire date: Feb 23 07:31:54 2024 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5637d6100700)
> GET / HTTP/2
> Host: details.mydomain.com
> user-agent: curl/7.78.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< date: Thu, 23 Feb 2023 09:18:07 GMT
< content-type: text/html
< content-length: 6
< last-modified: Thu, 23 Feb 2023 08:43:14 GMT
< etag: "63f72722-6"
< accept-ranges: bytes
< strict-transport-security: max-age=15724800; includeSubDomains
< 
index

and connection to HTTPS node port of nginx-ingress

curl -v -H 'Host: details.mydomain.com.com' https://10.0.0.246:31567 -k                           
*   Trying 10.0.0.246:31567...
* Connected to 10.0.0.246 (10.0.0.246) port 31567 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb 23 07:31:54 2023 GMT
*  expire date: Feb 23 07:31:54 2024 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: details.mydomain.com.com]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x13c011e00)
> GET / HTTP/2
> Host: details.mydomain.com.com
> user-agent: curl/7.86.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< date: Thu, 23 Feb 2023 09:25:23 GMT
< content-type: text/html
< content-length: 6
< last-modified: Thu, 23 Feb 2023 08:43:14 GMT
< etag: "63f72722-6"
< accept-ranges: bytes
< strict-transport-security: max-age=15724800; includeSubDomains
< 
index
* Connection #0 to host 10.0.0.246 left intact

when I do connection to AWS NLB address, I got 400 The plain HTTP request was sent to HTTPS port

$ curl -v -H 'Host: details.mydomain.com' https://a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.us-east-1.amazonaws.com -k
*   Trying 10.0.2.129:443...
* Connected to a0f5d92b767be4da8b7c203e4521ddfa-b44845ad0f2ecfa8.elb.us-east-1.amazonaws.com (10.0.2.129) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Aug 14 00:00:00 2022 GMT
*  expire date: Sep 12 23:59:59 2023 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: details.mydomain.com.com
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Thu, 23 Feb 2023 09:28:13 GMT
< Content-Type: text/html
< Content-Length: 248
< Connection: close
< 
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0
* TLSv1.2 (IN), TLS alert, close notify (256):
* TLSv1.2 (OUT), TLS alert, close notify (256):

It seems something wrong between AWS NLB and ingress pod. I will focus on NLB settings

ismailyenigul commented 1 year ago

ELB target group for TLS listener 443 was set as TCP: 31567 I changed to TLS: 31567 manually and now I can access to my url successfully. I will check nginx ingress controller to set LB target group protocol as TLS not TCP.

ismailyenigul commented 1 year ago

Finally!.. service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl was the magic annotation. it was service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp initially complete annotations:

service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-internal: true
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "100"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:xyz:certificate