search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.47k stars 8.25k forks source link

Ingress Controller returns 404 error for all service routes #8053

Closed Slyke closed 2 years ago

Slyke commented 2 years ago

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.1.0
  Build:         cacbee86b6ccc45bde8ffc184521bed3022e7dee
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

Environment:

The result of these commands are huge.

What happened:

Ingress controller returns 404 for all service routes.

What you expected to happen:

Should be able to navigate to mydomain.xyz/ and the URL be forwarded to the reverse proxy.

How to reproduce it:

To Reproduce Steps to reproduce the behavior:

  1. Install MetalLB
  2. Install cert-bot with nginx ingress controller
  3. Point the ingress controller to a nginx reverse proxy

Ingress for reverse proxy's service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: nginx-ingress
    namespace: cert-manager
  annotations:
    cert-manager.io/acme-challenge-type: http01
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prod"
    # nginx.ingress.kubernetes.io/rewrite-target: /
  name: nginx-ingress
  namespace: cert-manager
spec:
  rules:
    - host: mydomain.xyz
      http:
        paths:
        - path: /*
          pathType: Prefix
          backend:
            service:
              name: nginx-rp
              port:
                number: 80
  tls:
    - hosts:
      - mydomain.xyz
      secretName: mydomain.xyz-tls

My reverse proxy default.conf:

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;

  location / {
    add_header Content-Type text/plain;
    return 200 "Default route";
  }

  location /test {
    add_header Content-Type text/plain;
    return 200 "Test route";
  }
}

If nginx.ingress.kubernetes.io/rewrite-target: / is not set, the ingress controller returns 404. If it is set, then the ingress controller will forward the request to the downstream reverse proxy, but it will only forward what is explicitly set here. With it only /, it will always return Default route even when navigating to mydomain.xyz/test. I can cURL the downstream reverse proxy directly and see that it is returning the correct response for each route.

Anything else we need to know: Controller startup logs:

NGINX Ingress controller
  Release:       v1.1.0
  Build:         cacbee86b6ccc45bde8ffc184521bed3022e7dee
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

W1219 13:39:30.406196       8 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1219 13:39:30.406804       8 main.go:223] "Creating API client" host="https://10.96.0.1:443"
I1219 13:39:30.450479       8 main.go:267] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.1" state="clean" commit="632ed300f2c34f6d6d15ca4cef3d3c7073412212" platform="linux/amd64"
I1219 13:39:31.601570       8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1219 13:39:31.699169       8 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1219 13:39:31.772508       8 nginx.go:255] "Starting NGINX Ingress controller"
I1219 13:39:31.876384       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"cert-manager", Name:"quickstart-ingress-nginx-controller", UID:"fe09650f-c6f9-4003-9d78-205b800b0f16", APIVersion:"v1", ResourceVersion:"18026426", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap cert-manager/quickstart-ingress-nginx-controller
I1219 13:39:32.985217       8 store.go:424] "Found valid IngressClass" ingress="cert-manager/nginx-ingress" ingressclass="nginx"
W1219 13:39:32.985953       8 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "cert-manager/mydomain.xyz-tls" in local store
I1219 13:39:32.986021       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cert-manager", Name:"nginx-ingress", UID:"011ce722-c49b-4e78-b546-f9ecd9d2ebdc", APIVersion:"networking.k8s.io/v1", ResourceVersion:"18253999", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 13:39:33.075491       8 nginx.go:297] "Starting NGINX process"
I1219 13:39:33.075560       8 leaderelection.go:248] attempting to acquire leader lease cert-manager/ingress-controller-leader...
I1219 13:39:33.079703       8 nginx.go:317] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
W1219 13:39:33.079960       8 controller.go:1299] Error getting SSL certificate "cert-manager/mydomain.xyz-tls": local SSL certificate cert-manager/mydomain.xyz-tls was not found. Using default certificate
I1219 13:39:33.080054       8 controller.go:155] "Configuration changes detected, backend reload required"
I1219 13:39:33.092910       8 status.go:84] "New leader elected" identity="quickstart-ingress-nginx-controller-54f6f89679-mgrpk"
I1219 13:39:33.365905       8 controller.go:172] "Backend successfully reloaded"
I1219 13:39:33.366113       8 controller.go:183] "Initial sync, sleeping for 1 second"
I1219 13:39:33.366200       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"cert-manager", Name:"quickstart-ingress-nginx-controller-54f6f89679-778j6", UID:"7fdd7193-d239-41d7-acb5-e6f8c6ed01c9", APIVersion:"v1", ResourceVersion:"18263460", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1219 13:39:36.861614       8 controller.go:1299] Error getting SSL certificate "cert-manager/mydomain.xyz-tls": local SSL certificate cert-manager/mydomain.xyz-tls was not found. Using default certificate
W1219 13:39:40.196209       8 controller.go:1299] Error getting SSL certificate "cert-manager/mydomain.xyz-tls": local SSL certificate cert-manager/mydomain.xyz-tls was not found. Using default certificate

/kind bug

k8s-ci-robot commented 2 years ago

@Slyke: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 2 years ago

Based on above 2 bullets, it is best that you discuss this in the ingress-nginx-users channel on kubernetes.slack.com . There is no documented process here https://kubernetes.github.io/ingress-nginx/, that seems related to what you are attempting.

/remove-kind bug /kind support

Slyke commented 2 years ago

Hello, I figured out how to do this: https://kubernetes.github.io/ingress-nginx/examples/rewrite/

Was as easy as:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: your-ingress
    namespace: cert-manager
  annotations:
    cert-manager.io/acme-challenge-type: http01
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /$1 # Pass capture group here
  name: your-ingress
  namespace: cert-manager
spec:
  rules:
    - host: your-domain.com
      http:
        paths:
        - path: /(.*) # Setup capture group here
          pathType: Prefix
          backend:
            service:
              name: your-service
              port:
                number: 80
  tls:
    - hosts:
      - your-domain.com
      secretName: your-domain.com-tls-cert