search

nginxinc / kubernetes-ingress

NGINX and NGINX Plus Ingress Controllers for Kubernetes
https://docs.nginx.com/nginx-ingress-controller
Apache License 2.0
4.66k stars 1.97k forks source link

Ingress Controller returns 404 error for all service routes #2295

Closed Slyke closed 2 years ago

Slyke commented 2 years ago

Describe the bug Ingress controller returns 404 for all service routes. This might be related to #1274

To Reproduce Steps to reproduce the behavior:

  1. Install MetalLB
  2. Install cert-bot with nginx ingress controller
  3. Point the ingress controller to a nginx reverse proxy

Ingress for reverse proxy's service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: nginx-ingress
    namespace: cert-manager
  annotations:
    cert-manager.io/acme-challenge-type: http01
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prod"
    # nginx.ingress.kubernetes.io/rewrite-target: /
  name: nginx-ingress
  namespace: cert-manager
spec:
  rules:
    - host: mydomain.xyz
      http:
        paths:
        - path: /*
          pathType: Prefix
          backend:
            service:
              name: nginx-rp
              port:
                number: 80
  tls:
    - hosts:
      - mydomain.xyz
      secretName: mydomain.xyz-tls

My reverse proxy default.conf:

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;

  location / {
    add_header Content-Type text/plain;
    return 200 "Default route";
  }

  location /test {
    add_header Content-Type text/plain;
    return 200 "Test route";
  }
}

Expected behavior Should be able to navigate to mydomain.xyz/ and the URL `` be forwarded to the reverse proxy.

Release: v1.1.0 Build: cacbee86b6ccc45bde8ffc184521bed3022e7dee Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.9

Your environment

Additional context If nginx.ingress.kubernetes.io/rewrite-target: / is not set, the ingress controller returns 404. If it is set, then the ingress controller will forward the request to the downstream reverse proxy, but it will only forward what is explicitly set here. With it only /, it will always return Default route even when navigating to mydomain.xyz/test. I can cURL the downstream reverse proxy directly and see that it is returning the correct response for each route.

Controller startup logs:

NGINX Ingress controller
  Release:       v1.1.0
  Build:         cacbee86b6ccc45bde8ffc184521bed3022e7dee
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

W1219 13:39:30.406196       8 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1219 13:39:30.406804       8 main.go:223] "Creating API client" host="https://10.96.0.1:443"
I1219 13:39:30.450479       8 main.go:267] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.1" state="clean" commit="632ed300f2c34f6d6d15ca4cef3d3c7073412212" platform="linux/amd64"
I1219 13:39:31.601570       8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1219 13:39:31.699169       8 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1219 13:39:31.772508       8 nginx.go:255] "Starting NGINX Ingress controller"
I1219 13:39:31.876384       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"cert-manager", Name:"quickstart-ingress-nginx-controller", UID:"fe09650f-c6f9-4003-9d78-205b800b0f16", APIVersion:"v1", ResourceVersion:"18026426", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap cert-manager/quickstart-ingress-nginx-controller
I1219 13:39:32.985217       8 store.go:424] "Found valid IngressClass" ingress="cert-manager/nginx-ingress" ingressclass="nginx"
W1219 13:39:32.985953       8 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "cert-manager/mydomain.xyz-tls" in local store
I1219 13:39:32.986021       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cert-manager", Name:"nginx-ingress", UID:"011ce722-c49b-4e78-b546-f9ecd9d2ebdc", APIVersion:"networking.k8s.io/v1", ResourceVersion:"18253999", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 13:39:33.075491       8 nginx.go:297] "Starting NGINX process"
I1219 13:39:33.075560       8 leaderelection.go:248] attempting to acquire leader lease cert-manager/ingress-controller-leader...
I1219 13:39:33.079703       8 nginx.go:317] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
W1219 13:39:33.079960       8 controller.go:1299] Error getting SSL certificate "cert-manager/mydomain.xyz-tls": local SSL certificate cert-manager/mydomain.xyz-tls was not found. Using default certificate
I1219 13:39:33.080054       8 controller.go:155] "Configuration changes detected, backend reload required"
I1219 13:39:33.092910       8 status.go:84] "New leader elected" identity="quickstart-ingress-nginx-controller-54f6f89679-mgrpk"
I1219 13:39:33.365905       8 controller.go:172] "Backend successfully reloaded"
I1219 13:39:33.366113       8 controller.go:183] "Initial sync, sleeping for 1 second"
I1219 13:39:33.366200       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"cert-manager", Name:"quickstart-ingress-nginx-controller-54f6f89679-778j6", UID:"7fdd7193-d239-41d7-acb5-e6f8c6ed01c9", APIVersion:"v1", ResourceVersion:"18263460", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1219 13:39:36.861614       8 controller.go:1299] Error getting SSL certificate "cert-manager/mydomain.xyz-tls": local SSL certificate cert-manager/mydomain.xyz-tls was not found. Using default certificate
W1219 13:39:40.196209       8 controller.go:1299] Error getting SSL certificate "cert-manager/mydomain.xyz-tls": local SSL certificate cert-manager/mydomain.xyz-tls was not found. Using default certificate
github-actions[bot] commented 2 years ago

Hi @Slyke thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:

I've parsed the text of your issue and it looks like you might be mixing up the two Ingress Controllers, please take a look at this page to see the differences between nginxinc/kubernetes-ingress (this repo) and kubernetes/ingress-nginx.

Cheers!

jasonwilliams14 commented 2 years ago

@Slyke looks like you are using a different NGINX Ingress project. From your information above, you are using this project:

https://github.com/kubernetes/ingress-nginx

Which is different from this project. https://github.com/nginxinc/kubernetes-ingress

HTH!

Slyke commented 2 years ago

@jasonwilliams14 ooo sorry about that! I have posted it on the correct github https://github.com/kubernetes/ingress-nginx/issues/8053 . Will close this issue!