search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.03k stars 8.16k forks source link

Mixed content errors #10688

Open ebiscaia opened 8 months ago

ebiscaia commented 8 months ago

Hi all,

Sorry if this is not following the procedures of how to post my problem.

I am trying to self-host speedtest-tracker (https://github.com/alexjustesen/speedtest-tracker) and my system has a Raspberry Pi with Nginx Reverse Proxy and a VM with K0S, MetalLB and Nginx Ingress. The Raspberry Pi provides access to all my services (Proxmox, HA, ...) and passes to Nginx Ingress the services that are in Kubernetes. The problem is that with speed-tracker the pages are not loading correctly if I use ingress. I get the blocked:mixed-content)

Things I have tried so far: Fix the Reverse Proxy: I added proxy_set_header X-Forwarded-Proto $scheme; in the Raspberry Pi and create a Load Balancer service in my deployment and it worked. That actually was the initial configuration (normally I test things with a Load Balancer and then go to Ingress). Remove the Reverse Proxy: I removed the entry and created one in /etc/hosts. Worked. That was just to check if there was any other issues with the ingress and that was not the case. I returned to use the Reverse Proxy afterwards. Add annotations to the ingress of the application: nginx.ingress.kubernetes.io/proxy-set-header: "X-Forwarded-Proto $scheme": Did not work. nginx.ingress.kubernetes.io/force-ssl-redirect: "true": Makes the error: ERR_TOO_MANY_REDIRECTS Edit nginx cmap: Add use-forwarded-headers: true. Did not work

Here are the links for comparison: http://speedtest.eddienetworks.ddnsfree.com http://speedtest_lb.eddienetworks.ddnsfree.com (Temporary)

Thanks

k8s-ci-robot commented 8 months ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 8 months ago

We don't test with reverse-proxies . /remove-kind bug

Generally you will need to enbale proxy-protocol. If you post the answers to the questions that are asked in a new issue template, it may make someone comment usefully on that data. /triage needs-information

ebiscaia commented 8 months ago

Let's see if what I got is useful:

I apologise the long post but details seems to break the code block. Let me know how to fix it and I'll edit the post.

What happened: Page does not load css content due to mixed-errors. See http://speedtest.eddienetworks.ddnsfree.com

What you expect to happen: Mixed-errors not to happen. Page loads correctly. See http://speedtest_lb.eddienetworks.ddnsfree.com. Same application but in load balancer service type. Just for comparison.

NGINX Ingress controller version

NGINX Ingress controller
  Release:       v1.8.1
  Build:         dc88dce9ea5e700f3301d16f971fa17c6cfe757d
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Kubernetes

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3+k0s

Environment

OS: Alpine Linux v3.19 x86_64
Host: KVM/QEMU (Standard PC (Q35 + ICH9, 2009) pc-q35-8.1)
Kernel: 6.1.63-0-virt
CPU: QEMU Virtual version 2.5+ (4) @ 3.191GHz
Memory: 4934MiB

Instalation k0s

Cluster info

NAME      STATUS   ROLES           AGE   VERSION       INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
master    Ready    control-plane   83d   v1.28.3+k0s   192.168.1.170   <none>        Alpine Linux v3.19               6.1.63-0-virt      containerd://1.7.8
worker1   Ready    <none>          83d   v1.28.4+k0s   192.168.1.171   <none>        Ubuntu 23.10                     6.5.0-13-generic   containerd://1.7.8
worker2   Ready    <none>          83d   v1.28.4+k0s   192.168.1.172   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-13-amd64     containerd://1.7.8
worker3   Ready    <none>          79d   v1.28.4+k0s   192.168.1.173   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-13-amd64     containerd://1.7.8

How was the ingress-nginx-controller installed:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.1/deploy/static/provider/cloud/deploy.yaml

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.96.0.1 443/TCP 83d grafana service/grafana ClusterIP 10.108.247.33 3000/TCP 6d3h app=grafana guacamole service/guacamole-svc LoadBalancer 10.106.196.44 192.168.1.241 8080:30123/TCP 6d2h app=guacamole homer service/homer ClusterIP 10.108.16.142 5100/TCP 6d app=homer ingress-nginx service/ingress-nginx-controller LoadBalancer 10.111.24.64 192.168.1.240 80:32157/TCP,443:30789/TCP 82d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.103.106.2 443/TCP 82d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx jellyfin service/jellyfin ClusterIP 10.100.221.220 8111/TCP 80d app=jellyfin kube-system service/kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 83d k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.103.16.125 443/TCP 83d k8s-app=metrics-server kuma service/kuma ClusterIP 10.105.155.222 3001/TCP 6d8h app=kuma longhorn-system service/csi-attacher ClusterIP 10.97.240.180 12345/TCP 6d8h app=csi-attacher longhorn-system service/csi-provisioner ClusterIP 10.98.81.204 12345/TCP 6d8h app=csi-provisioner longhorn-system service/csi-resizer ClusterIP 10.104.30.233 12345/TCP 6d8h app=csi-resizer longhorn-system service/csi-snapshotter ClusterIP 10.103.57.95 12345/TCP 6d8h app=csi-snapshotter longhorn-system service/longhorn-admission-webhook ClusterIP 10.97.188.59 9502/TCP 81d app=longhorn-manager longhorn-system service/longhorn-backend ClusterIP 10.108.38.26 9500/TCP 81d app=longhorn-manager longhorn-system service/longhorn-conversion-webhook ClusterIP 10.103.225.213 9501/TCP 81d app=longhorn-manager longhorn-system service/longhorn-engine-manager ClusterIP None 81d longhorn.io/component=instance-manager,longhorn.io/instance-manager-type=engine longhorn-system service/longhorn-frontend ClusterIP 10.106.228.10 80/TCP 81d app=longhorn-ui longhorn-system service/longhorn-recovery-backend ClusterIP 10.106.241.56 9503/TCP 81d app=longhorn-manager longhorn-system service/longhorn-replica-manager ClusterIP None 81d longhorn.io/component=instance-manager,longhorn.io/instance-manager-type=replica longhorn-system service/pvc-16ee8b72-aeed-48da-85ff-0d6646ee5df8 ClusterIP 10.105.172.79 2049/TCP 6d longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-16ee8b72-aeed-48da-85ff-0d6646ee5df8 longhorn-system service/pvc-3d5fc6c2-294f-4ef7-a826-60068f858348 ClusterIP 10.100.192.81 2049/TCP 34d longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-3d5fc6c2-294f-4ef7-a826-60068f858348 longhorn-system service/pvc-5c6b76b8-b373-48b6-b88b-232bb6a3bc08 ClusterIP 10.103.90.215 2049/TCP 6d2h longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-5c6b76b8-b373-48b6-b88b-232bb6a3bc08 longhorn-system service/pvc-7ab5716c-33cc-4ac0-a327-73fa8e61d3f1 ClusterIP 10.103.185.154 2049/TCP 6d3h longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-7ab5716c-33cc-4ac0-a327-73fa8e61d3f1 longhorn-system service/pvc-9452b7a8-9511-459b-91db-1505f4d6852b ClusterIP 10.101.183.144 2049/TCP 6d8h longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-9452b7a8-9511-459b-91db-1505f4d6852b longhorn-system service/pvc-a0c2f323-a4a3-4832-adf3-ccc8be7afddf ClusterIP 10.102.37.0 2049/TCP 6d6h longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-a0c2f323-a4a3-4832-adf3-ccc8be7afddf longhorn-system service/pvc-c5602baf-d814-4f27-a21d-2baee3a787d7 ClusterIP 10.108.178.110 2049/TCP 3h10m longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-c5602baf-d814-4f27-a21d-2baee3a787d7 longhorn-system service/pvc-df002cf8-a820-4396-849e-65867dfae96e ClusterIP 10.102.43.249 2049/TCP 6d1h longhorn.io/managed-by=longhorn-manager,longhorn.io/share-manager=pvc-df002cf8-a820-4396-849e-65867dfae96e metallb-system service/webhook-service ClusterIP 10.100.224.5 443/TCP 82d component=controller navid service/navidrome LoadBalancer 10.105.244.72 192.168.1.243 5050:30957/TCP 34d app=navidrome piwigo service/mariadb-svc ClusterIP 10.107.117.67 3300/TCP 3d10h app=mariadb piwigo service/piwigo-svc LoadBalancer 10.102.233.180 192.168.1.246 3344:30806/TCP 3d10h app=piwigo podgrab service/podgrab-svc LoadBalancer 10.104.26.118 192.168.1.245 8080:31048/TCP 6d1h app=podgrab portainer service/portainer-agent LoadBalancer 10.106.214.29 192.168.1.242 9001:31446/TCP 66d app=portainer-agent portainer service/portainer-agent-headless ClusterIP None 66d app=portainer-agent prometheus service/prometheus ClusterIP 10.101.200.1 9090/TCP 6d6h app=prometheus speedtest service/mariadb-svc ClusterIP 10.103.125.252 3300/TCP 2d3h app=mariadb speedtest service/speedtest-svc LoadBalancer 10.107.231.121 192.168.1.247 3344:32165/TCP 2d2h app=speedtest vaultwarden service/vaultwarden LoadBalancer 10.99.43.37 192.168.1.244 5555:31761/TCP 9d app=vaultwarden

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/konnectivity-agent 4 4 4 4 4 kubernetes.io/os=linux 83d konnectivity-agent quay.io/k0sproject/apiserver-network-proxy-agent:v0.1.4 k8s-app=konnectivity-agent kube-system daemonset.apps/kube-proxy 4 4 4 4 4 kubernetes.io/os=linux 83d kube-proxy quay.io/k0sproject/kube-proxy:v1.28.3 k8s-app=kube-proxy kube-system daemonset.apps/kube-router 4 4 4 4 4 83d kube-router quay.io/k0sproject/kube-router:v1.6.0-iptables1.8.9-0 k8s-app=kube-router,tier=node longhorn-system daemonset.apps/engine-image-ei-74783864 3 3 3 3 3 81d engine-image-ei-74783864 longhornio/longhorn-engine:v1.5.1 longhorn.io/component=engine-image,longhorn.io/engine-image=ei-74783864 longhorn-system daemonset.apps/longhorn-csi-plugin 3 3 3 3 3 6d8h node-driver-registrar,longhorn-liveness-probe,longhorn-csi-plugin longhornio/csi-node-driver-registrar:v2.7.0,longhornio/livenessprobe:v2.9.0,longhornio/longhorn-manager:v1.5.1 app=longhorn-csi-plugin longhorn-system daemonset.apps/longhorn-manager 3 3 3 3 3 81d longhorn-manager longhornio/longhorn-manager:v1.5.1 app=longhorn-manager metallb-system daemonset.apps/speaker 4 4 4 4 4 kubernetes.io/os=linux 82d speaker quay.io/metallb/speaker:v0.13.10 app=metallb,component=speaker

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR grafana deployment.apps/grafana 1/1 1 1 6d3h grafana grafana/grafana-oss app=grafana guacamole deployment.apps/guacamole-dpl 1/1 1 1 6d2h guacamole flcontainers/guacamole app=guacamole homer deployment.apps/homer-dpl 3/3 3 3 6d doughnut b4bz/homer app=homer ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 82d controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx jellyfin deployment.apps/jellyfin 1/1 1 1 3h10m jellyfin lscr.io/linuxserver/jellyfin:latest app=jellyfin kube-system deployment.apps/coredns 2/2 2 2 83d coredns quay.io/k0sproject/coredns:1.11.1 k8s-app=kube-dns kube-system deployment.apps/metrics-server 1/1 1 1 83d metrics-server registry.k8s.io/metrics-server/metrics-server:v0.6.4 k8s-app=metrics-server kuma deployment.apps/kuma 1/1 1 1 6d8h kuma louislam/uptime-kuma app=kuma longhorn-system deployment.apps/csi-attacher 3/3 3 3 6d8h csi-attacher longhornio/csi-attacher:v4.2.0 app=csi-attacher longhorn-system deployment.apps/csi-provisioner 3/3 3 3 6d8h csi-provisioner longhornio/csi-provisioner:v3.4.1 app=csi-provisioner longhorn-system deployment.apps/csi-resizer 3/3 3 3 6d8h csi-resizer longhornio/csi-resizer:v1.7.0 app=csi-resizer longhorn-system deployment.apps/csi-snapshotter 3/3 3 3 6d8h csi-snapshotter longhornio/csi-snapshotter:v6.2.1 app=csi-snapshotter longhorn-system deployment.apps/longhorn-driver-deployer 1/1 1 1 81d longhorn-driver-deployer longhornio/longhorn-manager:v1.5.1 app=longhorn-driver-deployer longhorn-system deployment.apps/longhorn-ui 2/2 2 2 81d longhorn-ui longhornio/longhorn-ui:v1.5.1 app=longhorn-ui metallb-system deployment.apps/controller 1/1 1 1 82d controller quay.io/metallb/controller:v0.13.10 app=metallb,component=controller navid deployment.apps/navidrome 1/1 1 1 4d1h navi deluan/navidrome:0.49.2 app=navidrome piwigo deployment.apps/mariadb-dpl 1/1 1 1 3d1h mariadb linuxserver/mariadb app=mariadb piwigo deployment.apps/piwigo 1/1 1 1 2d6h piwigo linuxserver/piwigo app=piwigo podgrab deployment.apps/podgrab 1/1 1 1 6d1h podgrab akhilrex/podgrab app=podgrab portainer deployment.apps/portainer-agent 1/1 1 1 66d portainer-agent portainer/agent:2.19.1 app=portainer-agent prometheus deployment.apps/prometheus 1/1 1 1 6d6h prometheus prom/prometheus app=prometheus speedtest deployment.apps/mariadb-dpl 1/1 1 1 2d2h mariadb linuxserver/mariadb app=mariadb speedtest deployment.apps/speedtest 1/1 1 1 163m speedtest ghcr.io/alexjustesen/speedtest-tracker:v0.13.3 app=speedtest vaultwarden deployment.apps/vaultwarden 1/1 1 1 7d20h vaultwarden vaultwarden/server app=vaultwarden

NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR grafana replicaset.apps/grafana-64bd4b4ff 1 1 1 6d3h grafana grafana/grafana-oss app=grafana,pod-template-hash=64bd4b4ff guacamole replicaset.apps/guacamole-dpl-d9c674794 1 1 1 6d2h guacamole flcontainers/guacamole app=guacamole,pod-template-hash=d9c674794 homer replicaset.apps/homer-dpl-5d7d6576b6 3 3 3 6d doughnut b4bz/homer app=homer,pod-template-hash=5d7d6576b6 ingress-nginx replicaset.apps/ingress-nginx-controller-6858cb9dd9 0 0 0 33m controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6858cb9dd9 ingress-nginx replicaset.apps/ingress-nginx-controller-79d66f886c 0 0 0 82d controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=79d66f886c ingress-nginx replicaset.apps/ingress-nginx-controller-7f6c4db675 0 0 0 68m controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7f6c4db675 ingress-nginx replicaset.apps/ingress-nginx-controller-8466cbd75b 1 1 1 28m controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=8466cbd75b jellyfin replicaset.apps/jellyfin-7994d8b8c 1 1 1 3h10m jellyfin lscr.io/linuxserver/jellyfin:latest app=jellyfin,pod-template-hash=7994d8b8c kube-system replicaset.apps/coredns-85df575cdb 2 2 2 11d coredns quay.io/k0sproject/coredns:1.11.1 k8s-app=kube-dns,pod-template-hash=85df575cdb kube-system replicaset.apps/coredns-878bb57ff 0 0 0 83d coredns quay.io/k0sproject/coredns:1.10.1 k8s-app=kube-dns,pod-template-hash=878bb57ff kube-system replicaset.apps/metrics-server-7cdb99bf49 1 1 1 83d metrics-server registry.k8s.io/metrics-server/metrics-server:v0.6.4 k8s-app=metrics-server,pod-template-hash=7cdb99bf49 kuma replicaset.apps/kuma-76756967fc 1 1 1 6d8h kuma louislam/uptime-kuma app=kuma,pod-template-hash=76756967fc longhorn-system replicaset.apps/csi-attacher-7b5979f545 3 3 3 6d8h csi-attacher longhornio/csi-attacher:v4.2.0 app=csi-attacher,pod-template-hash=7b5979f545 longhorn-system replicaset.apps/csi-provisioner-55d544784d 3 3 3 6d8h csi-provisioner longhornio/csi-provisioner:v3.4.1 app=csi-provisioner,pod-template-hash=55d544784d longhorn-system replicaset.apps/csi-resizer-5bd864fbf6 3 3 3 6d8h csi-resizer longhornio/csi-resizer:v1.7.0 app=csi-resizer,pod-template-hash=5bd864fbf6 longhorn-system replicaset.apps/csi-snapshotter-8dcd84758 3 3 3 6d8h csi-snapshotter longhornio/csi-snapshotter:v6.2.1 app=csi-snapshotter,pod-template-hash=8dcd84758 longhorn-system replicaset.apps/longhorn-driver-deployer-96cb874b9 1 1 1 81d longhorn-driver-deployer longhornio/longhorn-manager:v1.5.1 app=longhorn-driver-deployer,pod-template-hash=96cb874b9 longhorn-system replicaset.apps/longhorn-ui-67bfdc7cf9 2 2 2 81d longhorn-ui longhornio/longhorn-ui:v1.5.1 app=longhorn-ui,pod-template-hash=67bfdc7cf9 metallb-system replicaset.apps/controller-595f88d88f 1 1 1 82d controller quay.io/metallb/controller:v0.13.10 app=metallb,component=controller,pod-template-hash=595f88d88f navid replicaset.apps/navidrome-794f456777 1 1 1 4d1h navi deluan/navidrome:0.49.2 app=navidrome,pod-template-hash=794f456777 piwigo replicaset.apps/mariadb-dpl-5cfd67bb86 1 1 1 3d1h mariadb linuxserver/mariadb app=mariadb,pod-template-hash=5cfd67bb86 piwigo replicaset.apps/piwigo-65bccc5b4f 1 1 1 2d6h piwigo linuxserver/piwigo app=piwigo,pod-template-hash=65bccc5b4f podgrab replicaset.apps/podgrab-6c58d7b5f6 1 1 1 6d1h podgrab akhilrex/podgrab app=podgrab,pod-template-hash=6c58d7b5f6 portainer replicaset.apps/portainer-agent-7b78fd9984 1 1 1 66d portainer-agent portainer/agent:2.19.1 app=portainer-agent,pod-template-hash=7b78fd9984 prometheus replicaset.apps/prometheus-6684c8d569 1 1 1 6d6h prometheus prom/prometheus app=prometheus,pod-template-hash=6684c8d569 speedtest replicaset.apps/mariadb-dpl-86d69ff55d 1 1 1 2d2h mariadb linuxserver/mariadb app=mariadb,pod-template-hash=86d69ff55d speedtest replicaset.apps/speedtest-6d67c8b45 1 1 1 154m speedtest ghcr.io/alexjustesen/speedtest-tracker:v0.13.3 app=speedtest,pod-template-hash=6d67c8b45 speedtest replicaset.apps/speedtest-6fd54665f8 0 0 0 158m speedtest ghcr.io/alexjustesen/speedtest-tracker:v0.13.4 app=speedtest,pod-template-hash=6fd54665f8 speedtest replicaset.apps/speedtest-78bb6b566 0 0 0 163m speedtest ghcr.io/alexjustesen/speedtest-tracker app=speedtest,pod-template-hash=78bb6b566 speedtest replicaset.apps/speedtest-8489d978c7 0 0 0 158m speedtest ghcr.io/alexjustesen/speedtest-tracker:latest app=speedtest,pod-template-hash=8489d978c7 vaultwarden replicaset.apps/vaultwarden-54757947f5 0 0 0 7d20h vaultwarden vaultwarden/server app=vaultwarden,pod-template-hash=54757947f5 vaultwarden replicaset.apps/vaultwarden-5f494849 1 1 1 7d20h vaultwarden vaultwarden/server app=vaultwarden,pod-template-hash=5f494849

NAMESPACE NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR ingress-nginx job.batch/ingress-nginx-admission-create 1/1 24s 82d create registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b batch.kubernetes.io/controller-uid=4469c8e2-6709-48a1-8cc5-3c43425196d4 ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 26s 82d patch registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b batch.kubernetes.io/controller-uid=063a78bd-4711-4615-b234-509c5bfd98ff


  - `kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>`

Name: ingress-nginx-controller-8466cbd75b-lw2sg Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: worker2/192.168.1.172 Start Time: Thu, 30 Nov 2023 19:21:46 +1100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.8.1 pod-template-hash=8466cbd75b Annotations: Status: Running IP: 10.244.2.206 IPs: IP: 10.244.2.206 Controlled By: ReplicaSet/ingress-nginx-controller-8466cbd75b Containers: controller: Container ID: containerd://fedda818e2cd0ef889dc29525ec93d61524cba47b60fde945e1817f053f6f3f7 Image: registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd Image ID: registry.k8s.io/ingress-nginx/controller@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --v=3 State: Running Started: Thu, 30 Nov 2023 19:22:10 +1100 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-8466cbd75b-lw2sg (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xznpc (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-xznpc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 38m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-8466cbd75b-lw2sg to worker2 Normal Pulled 38m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd" already present on machine Normal Created 38m kubelet Created container controller Normal Started 38m kubelet Started container controller Normal RELOAD 38m nginx-ingress-controller NGINX reload triggered due to a change in configuration

  - `kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>`

Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.8.1 Annotations: metallb.universe.tf/ip-allocated-from-pool: pool Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.111.24.64 IPs: 10.111.24.64 LoadBalancer Ingress: 192.168.1.240 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32157/TCP Endpoints: 10.244.2.206:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30789/TCP Endpoints: 10.244.2.206:443 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 32304 Events: Type Reason Age From Message

Normal nodeAssigned 39m (x23 over 156m) metallb-speaker announcing from node "worker2" with protocol "layer2"

- **Current state of ingress object, if applicable**:
  - `kubectl -n <appnnamespace> get all,ing -o wide`

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/mariadb-dpl-86d69ff55d-q5r59 1/1 Running 2 (4h10m ago) 4h48m 10.244.2.184 worker2 pod/speedtest-6d67c8b45-7zqjc 1/1 Running 0 61m 10.244.2.203 worker2

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/mariadb-svc ClusterIP 10.103.125.252 3300/TCP 2d3h app=mariadb service/speedtest-svc LoadBalancer 10.107.231.121 192.168.1.247 3344:32165/TCP 2d2h app=speedtest

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/mariadb-dpl 1/1 1 1 2d3h mariadb linuxserver/mariadb app=mariadb deployment.apps/speedtest 1/1 1 1 177m speedtest ghcr.io/alexjustesen/speedtest-tracker:v0.13.3 app=speedtest

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/mariadb-dpl-86d69ff55d 1 1 1 2d3h mariadb linuxserver/mariadb app=mariadb,pod-template-hash=86d69ff55d replicaset.apps/speedtest-6d67c8b45 1 1 1 168m speedtest ghcr.io/alexjustesen/speedtest-tracker:v0.13.3 app=speedtest,pod-template-hash=6d67c8b45 replicaset.apps/speedtest-6fd54665f8 0 0 0 172m speedtest ghcr.io/alexjustesen/speedtest-tracker:v0.13.4 app=speedtest,pod-template-hash=6fd54665f8 replicaset.apps/speedtest-78bb6b566 0 0 0 177m speedtest ghcr.io/alexjustesen/speedtest-tracker app=speedtest,pod-template-hash=78bb6b566 replicaset.apps/speedtest-8489d978c7 0 0 0 173m speedtest ghcr.io/alexjustesen/speedtest-tracker:latest app=speedtest,pod-template-hash=8489d978c7

NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/speedtest-ing nginx speedtest.eddienetworks.ddnsfree.com 192.168.1.240 80 2d2h

  - `kubectl -n <appnamespace> describe ing <ingressname>`

Name: speedtest-ing Labels: Namespace: speedtest Address: 192.168.1.240 Ingress Class: nginx Default backend: Rules: Host Path Backends

speedtest.eddienetworks.ddnsfree.com
/ speedtest-svc:3344 (10.244.2.203:80) Annotations: Events: Type Reason Age From Message

Normal Sync 84m nginx-ingress-controller Scheduled for sync Normal Sync 65m nginx-ingress-controller Scheduled for sync Normal Sync 50m nginx-ingress-controller Scheduled for sync Normal Sync 45m nginx-ingress-controller Scheduled for sync Normal Sync 44m nginx-ingress-controller Scheduled for sync

  - If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

curl https://speedtest.eddienetworks.ddnsfree.com -v

Others nginx config of the reverse proxy:

upstream stest{
   server 192.168.1.240;
}

server {
   listen 80;
   server_name speedtest.eddienetworks.ddnsfree.com *.speedtest.eddienetworks.ddnsfree.com;
   return 301 https://$host$request_uri;
}

server {
   server_name speedtest.eddienetworks.ddnsfree.com *.speedtest.eddienetworks.ddnsfree.com;

   location / {
      proxy_pass http://stest/;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
   }

   listen [::]:443 ssl; # managed by Certbot
   listen 443 ssl; # managed by Certbot
   ssl_certificate /etc/letsencrypt/live/eddienetworks.ddnsfree.com/fullchain.pem; # managed by Certbot
   ssl_certificate_key /etc/letsencrypt/live/eddienetworks.ddnsfree.com/privkey.pem; # managed by Certbot
   include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Also from the ingress logs:

location / {
        set $namespace      "speedtest";
        set $ingress_name   "speedtest-ing";
        set $service_name   "speedtest-svc";
        set $service_port   "3344";
        set $location_path  "/";
        set $global_rate_limit_exceeding n;

        rewrite_by_lua_block {
            lua_ingress.rewrite({
                force_ssl_redirect = false,
                ssl_redirect = true,
                force_no_ssl_redirect = false,
                preserve_trailing_slash = false,
                use_port_in_redirects = false,
                global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
            })
            balancer.rewrite()
            plugins.run()
        }

        # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
        # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
        # other authentication method such as basic auth or external auth useless - all requests will be allowed.
        #access_by_lua_block {
        #}

        header_filter_by_lua_block {
            lua_ingress.header()
            plugins.run()
        }

        body_filter_by_lua_block {
            plugins.run()
        }

        log_by_lua_block {
            balancer.log()

            monitor.call()

            plugins.run()
        }

        port_in_redirect off;

        set $balancer_ewma_score -1;
        set $proxy_upstream_name "speedtest-speedtest-svc-3344";
        set $proxy_host          $proxy_upstream_name;
        set $pass_access_scheme  $scheme;

        set $pass_server_port    $server_port;

        set $best_http_host      $http_host;
        set $pass_port           $pass_server_port;

        set $proxy_alternative_upstream_name "";

        client_max_body_size                    1m;

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;

        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Request-ID           $req_id;
        proxy_set_header X-Real-IP              $remote_addr;

        proxy_set_header X-Forwarded-For        $remote_addr;

        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
        proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

        proxy_set_header X-Scheme               $pass_access_scheme;

        # Pass the original X-Forwarded-For
        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers to proxied server

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_buffering                         off;
        proxy_buffer_size                       4k;
        proxy_buffers                           4 4k;

        proxy_max_temp_file_size                1024m;

        proxy_request_buffering                 on;
        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

        # In case of errors try the next upstream server before returning an error
        proxy_next_upstream                     error timeout;
        proxy_next_upstream_timeout             0;
        proxy_next_upstream_tries               3;

        proxy_pass http://upstream_balancer;

        proxy_redirect                          off;

    }

}
## end server speedtest.eddienetworks.ddnsfree.com

Thanks for the attention

ebiscaia commented 7 months ago

Just to test, I installed a k3s VM (as k3s comes with Traefik ingress). The problem also happens in there.