search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.22k stars 8.2k forks source link

SSL not working with stream-snippet #9905

Open yaroslav-nakonechnikov opened 1 year ago

yaroslav-nakonechnikov commented 1 year ago

What happened:

to pass some tcp traffic i put that snippet:

 ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem; 
        ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem; 
        server { 
        listen 9997 ssl; 
        ssl_certificate_by_lua_block {      certificate.call() }  
        proxy_pass backend:9997; 
        }

by nginx documentation, ssl is allowed. in ingress it is failing with:

023/04/26 12:37:55 [error] 25#25: *57437 lua entry thread aborted: runtime error: ssl_certificate_by_lua:1: attemptto index global 'certificate' (a nil value)
stack traceback:

What you expected to happen:

something missing in lua script, so it waits for some info, which is missing in stream section.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.7.0
  Build:         72ff21ed9e26cb969052c753633049ba8a87ecf9
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26+", GitVersion:"v1.26.2-eks-a59e1f0", GitCommit:"8b68f4b95d7121d039ceebd30870e48acc7772e4", GitTreeState:"clean", BuildDate:"2023-03-09T19:59:45Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}

Environment:

after installation:

Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.7.0 helm.sh/chart=ingress-nginx-4.6.0 Annotations: ingressclass.kubernetes.io/is-default-class: true meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress Controller: k8s.io/ingress-nginx Events: 

  - `kubectl -n ingress get all -o wide`

root@ip-10-216-35-86 /]# kubectl -n ingress get all -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/ingress-nginx-controller-55cdbfc54c-btx2b 1/1 Running 0 4h41m 100.65.0.93 ip-100-65-1-227.eu-central-1.compute.internal

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller LoadBalancer 172.20.70.230 aff7d8ae25bd046dabd83d6454122237-bd390d2d2a32ef4c.elb.eu-central-1.amazonaws.com 443:32480/TCP,8089:30181/TCP,9997:31997/TCP 9d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller-admission ClusterIP 172.20.33.149 443/TCP 9d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 1/1 1 1 25h controller artifacts.domain/registry.k8s.io-docker-proxy/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-55cdbfc54c 1 1 1 25h controller artifacts.domain/registry.k8s.io-docker-proxy/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=55cdbfc54c replicaset.apps/ingress-nginx-controller-7b7b87c67f 0 0 0 7h10m controller artifacts.domain/registry.k8s.io-docker-proxy/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7b7b87c67f

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/ingress-nginx-controller Deployment/ingress-nginx-controller 20%/80%, 1%/80% 1 11 1 9d

  - `kubectl -n ingress describe po ingress-nginx-controller-55cdbfc54c-btx2b`

[root@ip-10-216-35-86 /]# kubectl -n ingress describe po ingress-nginx-controller-55cdbfc54c-btx2b Name: ingress-nginx-controller-55cdbfc54c-btx2b Namespace: ingress Priority: 0 Service Account: ingress-nginx Node: ip-100-65-1-227.eu-central-1.compute.internal/100.65.1.227 Start Time: Wed, 26 Apr 2023 11:13:07 +0000 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.7.0 helm.sh/chart=ingress-nginx-4.6.0 pod-template-hash=55cdbfc54c Annotations: Status: Running IP: 100.65.0.93 IPs: IP: 100.65.0.93 Controlled By: ReplicaSet/ingress-nginx-controller-55cdbfc54c Containers: controller: Container ID: containerd://b69679a15a7d71aeef518e01d3fcac4026b2aa35101a1e5af7968fa7d358e02e Image: artifacts.domain/registry.k8s.io-docker-proxy/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 Image ID: artifacts.domain/registry.k8s.io-docker-proxy/ingress-nginx/controller@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 Ports: 80/TCP, 443/TCP, 8443/TCP, 8089/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Wed, 26 Apr 2023 11:13:08 +0000 Ready: True Restart Count: 0 Requests: cpu: 128m memory: 512Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-55cdbfc54c-btx2b (v1:metadata.name) POD_NAMESPACE: ingress (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzssq (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-dzssq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: 

  - `kubectl -n ingress describe svc ingress-nginx-controller`

[root@ip-10-216-35-86 /]# kubectl -n ingress describe svc ingress-nginx-controller Name: ingress-nginx-controller Namespace: ingress Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.7.0 helm.sh/chart=ingress-nginx-4.6.0 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 60 service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-name: eks-31018 service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance service.beta.kubernetes.io/aws-load-balancer-scheme: internal service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-0fd26432879bc21c4,subnet-0ba383f86c9453911,subnet-0ebb77e75a9904da2 service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true service.beta.kubernetes.io/aws-load-balancer-target-node-labels: eks.amazonaws.com/nodegroup=eks-31018-default-2023041709535070060000001d service.beta.kubernetes.io/aws-load-balancer-type: nlb Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 172.20.70.230 IPs: 172.20.70.230 LoadBalancer Ingress: aff7d8ae25bd046dabd83d6454122237-bd390d2d2a32ef4c.elb.eu-central-1.amazonaws.com Port: https 443/TCP TargetPort: https/TCP NodePort: https 32480/TCP Endpoints: 100.65.0.93:443 Port: 8089-tcp 8089/TCP TargetPort: 8089-tcp/TCP NodePort: 8089-tcp 30181/TCP Endpoints: 100.65.0.93:8089 Port: stream 9997/TCP TargetPort: 9997/TCP NodePort: stream 31997/TCP Endpoints: 100.65.0.93:9997 Session Affinity: None External Traffic Policy: Cluster Events: 


- **Others**:
  - Any other related information like ;
    - copy/paste of the snippet (if applicable)

ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem; ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem; server { listen 9997 ssl; ssl_certificate_by_lua_block { certificate.call() }
proxy_pass backend:9997; }



**How to reproduce this issue**:
i used next docs to create setup:
- https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/
- https://nginx.org/en/docs/stream/ngx_stream_core_module.html
- https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream

And at least i'd like to know, where i can get on pod's level certificates, which are being used for https connection? 
generic usage of HTTPS works absolutely as expected.
k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 1 year ago

/remov-kind bug

These 2 links are from NGINX INC. company and not from this project ; https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/ https://nginx.org/en/docs/stream/ngx_stream_core_module.html

I think the TCP port provisioning functionality in this project is at layer 3/4 so and TLS/HTTPS is layer7 . But I could be completely wrong. Wait for comments from others.

You can create a ingress with TLS spec and look in the nginx.conf inside the controller pod to know about the TLS config.

longwuyuan commented 1 year ago

/remove-kind bug

yaroslav-nakonechnikov commented 1 year ago

@longwuyuan thank you, i've rechecked TLS configuration, and even added default-ssl-certificate to helm config. i see that this is applied in controller configuration, but not pod level, nginx.conf still same. SSL is being rendered by lua block, and this is an answer.

but, i believe i know how to implement workaround, with mounting secret additionally.

longwuyuan commented 1 year ago

Glad if you solved your problem. Please close the issue if issue is resolved.

I think that all TLS implemention goes into server blocks (including the default ssl cert) because that is still layer7 inspection, unlike the TCP port traffic.

yaroslav-nakonechnikov commented 1 year ago

problem is not solved. ingress uses lua to add certificates, and it doesn't support ssl for streams. and it is expected that streams will also support at least default certs if it is specified.

github-actions[bot] commented 1 year ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.