nginxinc / kubernetes-ingress

NGINX and NGINX Plus Ingress Controllers for Kubernetes
https://docs.nginx.com/nginx-ingress-controller
Apache License 2.0
4.64k stars 1.96k forks source link

Nginx Ingress Controller gives 404 not found #3436

Closed mohanss08 closed 1 year ago

mohanss08 commented 1 year ago
### Cluster information:
Kubernetes version:1.24.9
Bare-Metal being used:  (One master and three workers)
Installation method: Installed using Ansible.
Host OS: Ubuntu 18.04.6 LTS
CNI and version: Flannel ( rancher/mirrored-flannelcni-flannel - v0.19.2 )
CRI and version:  containerd://1.4.6

I have installed Nginx Ingress Controller by using this link https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests

From the above link. i have executed the below commands.

$ git clone https://github.com/nginxinc/kubernetes-ingress.git --branch v3.0.0
$ cd kubernetes-ingress/deployments
$ kubectl apply -f common/ns-and-sa.yaml
$ kubectl apply -f rbac/rbac.yaml

$ kubectl apply -f common/default-server-secret.yaml

$ kubectl apply -f common/nginx-config.yaml
$ kubectl apply -f common/ingress-class.yaml

$ kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
$ kubectl apply -f common/crds/k8s.nginx.org_policies.yaml

$ kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml

$ kubectl apply -f daemon-set/nginx-ingress.yaml

It is runs as a daemon set,

$ kubectl get all -n nginx-ingress
NAME                      READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-69z69   1/1     Running   0          26m
pod/nginx-ingress-l7z86   1/1     Running   0          26m
pod/nginx-ingress-wsjfv   1/1     Running   0          26m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress   3         3         3       3            3           <none>          26m
$ kubectl get svc  --all-namespaces
NAMESPACE     NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP                  3d1h
jenkins       jenkins-svc   ClusterIP   10.103.54.142   <none>        80/TCP                   3d1h
kube-system   kube-dns      ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   3d1h
$ kubectl get ingress --all-namespaces
NAMESPACE   NAME              CLASS    HOSTS                ADDRESS   PORTS   AGE
jenkins     jenkins-ingress   <none>   jenkins.company.com             80      19h
$ kubectl describe ingress jenkins-ingress -n jenkins
Name:             jenkins-ingress
Labels:           <none>
Namespace:        jenkins
Address:
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host             Path  Backends
  ----             ----  --------
  jenkins.company.com
                   /jenkins   jenkins-svc:80 (10.244.2.19:8080)
Annotations:       nginx.ingress.kubernetes.io/rewrite-target: /$2
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  9m4s  nginx-ingress-controller  Configuration for jenkins/jenkins-ingress was added or updated
  Normal  AddedOrUpdated  9m4s  nginx-ingress-controller  Configuration for jenkins/jenkins-ingress was added or updated
  Normal  AddedOrUpdated  9m4s  nginx-ingress-controller  Configuration for jenkins/jenkins-ingress was added or updated
$ kubectl exec -it -n nginx-ingress nginx-ingress-l7z86 -- curl -v -H "Host: nginx" localhost:80
*   Trying ::1:80...
* Connected to localhost (::1) port 80 (#0)
> GET / HTTP/1.1
> Host: nginx
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: nginx/1.23.3
< Date: Fri, 13 Jan 2023 11:06:53 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
<
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>
* Connection #0 to host localhost left intact

My Jenkins service yaml file as follows,

apiVersion: v1
kind: Service
metadata:
  name: jenkins-svc
  namespace: jenkins
  labels:
    app: jenkins
spec:
  selector: 
    app: jenkins
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

My ingress-resource.yaml file as follows

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: jenkins-ingress
  namespace: jenkins
spec:
  ingressClassName: nginx
  rules:
  - host: jenkins.company.com
    http:
      paths:
      - path: /jenkins
        pathType: Prefix
        backend:
          service:
            name: jenkins-svc
            port: 
              number: 80

When i tried to access our jenkins URL http://jenkins.company.com/jenkins getting 404 Not Found error on browser.

What could the problem? Also let me know Am i missing any additional installation commands?

github-actions[bot] commented 1 year ago

Hi @mohanss08 thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:

I've parsed the text of your issue and it looks like you might be mixing up the two Ingress Controllers, please take a look at this page to see the differences between nginxinc/kubernetes-ingress (this repo) and kubernetes/ingress-nginx.

Cheers!

shaun-nx commented 1 year ago

Thanks for all the details Mohan. I'm looking into this now and will get back to you asap.

shaun-nx commented 1 year ago

Hi Mohan. When you access the Jenkins URL http://jenkins.company.com/jenkins what result are you expecting to see? Do you have a Jenkins application deployed as a pod in your cluster?

From what I see you have service deployed but I don't see any pods or deployments for the service.

mohanss08 commented 1 year ago

Hi Shan,

When i access of my Jenkins URL - http://jenkins.company.com/jenkins i should see my Jenkins login page.

Yes i have deployed Jenkins application as a pod inside my cluster. Sorry in the question section i missed to include my application status.

So here it is.

$ kubectl get all -n jenkins
NAME                           READY   STATUS    RESTARTS   AGE
pod/jenkins-75cbc954b6-zph9f   1/1     Running   0          3d5h

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/jenkins-svc   ClusterIP   10.103.54.142   <none>        80/TCP    3d5h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jenkins   1/1     1            1           3d5h

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/jenkins-75cbc954b6   1         1         1       3d5h

For your information: My Jenkins application volume is PersistentVolume - NFS and the same is mounted in all hosts(Master & workers).

shaun-nx commented 1 year ago

Thank for the updated post Mohan. Can you share your deployment yaml file for deployment.apps/jenkins as well please?

mohanss08 commented 1 year ago

Thanks Shaun for assisting me. My deployment.yml file as follows.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
  labels:
    app: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      securityContext:
        fsGroup: 0 
        runAsUser: 0
      serviceAccountName: admin
      containers:
      - name: jenkins
        image: jenkins/jenkins:2.375.2-lts
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
          runAsUser: 0
        ports:
        - containerPort: 8080
        - containerPort: 50000
        env:
        - name: JENKINS_OPTS
          value: "--prefix=/jenkins --httpListenAddress=0.0.0.0"   
        - name: PROVISIONER_NAME
          value: app-nfs  
        - name: NFS_SERVER
          value: 165.13.45.56
        - name: NFS_PATH
          value: /app-nfs/kdata/jenkins
        volumeMounts:
          - name: jenkins-data
            mountPath: /var/jenkins_home         
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
            claimName: jenkins-pvc-nfs
shaun-nx commented 1 year ago

Hi Mohan. With the deployment files that you shared I have not been able to replicate the issue. Can you please run this command and share the output of it? This will print your full nginx config for your deployment. It should help us better understand what the problem is.

kubectl exec -it -n nginx-ingress nginx-ingress-l7z86 -- nginx -T

mohanss08 commented 1 year ago

Hi Shaun,

Requested output as follows.

$ kubectl exec -it -n nginx-ingress nginx-ingress-l7z86 -- nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

worker_processes  auto;
daemon off;

error_log  stderr notice;
pid        /var/lib/nginx/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    map $upstream_trailer_grpc_status $grpc_status {
        default $upstream_trailer_grpc_status;
        '' $sent_http_grpc_status;
    }

    access_log  /dev/stdout  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 65s;
    keepalive_requests 100;

    #gzip  on;

    server_names_hash_max_size 1024;
    server_names_hash_bucket_size 256;

    variables_hash_bucket_size 256;
    variables_hash_max_size 1024;

    map $request_uri $request_uri_no_args {
        "~^(?P<path>[^?]*)(\?.*)?$" $path;
    }

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    map $http_upgrade $vs_connection_header {
        default upgrade;
        ''      $default_connection_header;
    }

    server {
        # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
        set $default_connection_header "";
        set $resource_type "";
        set $resource_name "";
        set $resource_namespace "";
        set $service "";

        listen 80 default_server;
        listen [::]:80 default_server;

        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;

        ssl_certificate /etc/nginx/secrets/default;
        ssl_certificate_key /etc/nginx/secrets/default;

        server_name _;
        server_tokens "on";

        location / {
            return 404;
        }
    }
    # stub_status
    server {
        listen 8080;
        listen [::]:8080;

        allow 127.0.0.1;
        allow ::1;

        deny all;

        location /stub_status {
            stub_status;
        }
    }

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;

    server {
        listen unix:/var/lib/nginx/nginx-502-server.sock;
        access_log off;

        return 502;
    }

    server {
        listen unix:/var/lib/nginx/nginx-418-server.sock;
        access_log off;

        return 418;
    }
}

stream {
    log_format  stream-main  '$remote_addr [$time_local] '
                      '$protocol $status $bytes_sent $bytes_received '
                      '$session_time "$ssl_preread_server_name"';

    access_log  /dev/stdout  stream-main;

    map_hash_max_size 2048;
    map_hash_bucket_size 256;

    include /etc/nginx/stream-conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/config-version.conf:
server {
    listen unix:/var/lib/nginx/nginx-config-version.sock;
        access_log off;

    location /configVersion {
        return 200 2;
    }
}
map $http_x_expected_config_version $config_version_mismatch {
        "2" "";
        default "mismatch";
}
# configuration file /etc/nginx/conf.d/jenkins-jenkins-ingress.conf:
# configuration for jenkins/jenkins-ingress

upstream jenkins-jenkins-ingress-jenkins.company.com-jenkins-svc-80 {
        zone jenkins-jenkins-ingress-jenkins.company.com-jenkins-svc-80 256k;
        random two least_conn;

        server 10.244.2.2:8080 max_fails=1 fail_timeout=10s max_conns=0;

}

server {

        listen 80;
        listen [::]:80;

        server_tokens on;

        server_name jenkins.company.com;

        set $resource_type "ingress";
        set $resource_name "jenkins-ingress";
        set $resource_namespace "jenkins";

        location /jenkins {
                set $service "jenkins-svc";

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://jenkins-jenkins-ingress-jenkins.company.com-jenkins-svc-80;

        }

}
mohanss08 commented 1 year ago

Hi Shaun, Your inputs will be helpful, So is there any update on this issue?

shaun-nx commented 1 year ago

Hi Mohan. I tested with your configuration and everything looks to be fine. The problem seems to be with how you are trying to access the login page for Jenkins.

Since this is a local deployment your k8s cluster isn't able to resolve jenkins.company.com without deploying an external resolver.

I was able to access the login page by port-forwarding port 8080 of the jenkins pod. Example command: kubectl port-forward jenkins-6dd86fb567-w2r2q -n jenkins 8080:8080

You can then use http://127.0.0.1:8080/jenkins in your browser to access the login page. Example screenshot: image

Note: if you want to use the host name jenkins.company.com instead of 127.0.0.1 you will need to map the hostname to this IP in you /etc/hosts file.

mohanss08 commented 1 year ago

Hi Shaun, I'm using HA-Proxy (An Independent server) and Kubernetes cluster (One master and three workers) bare metal based.

My Access Flow Diagram:-

image

I have deployed Jenkins and Nginx ingress controller with help of above specified Link (Installation-with-manifests) method.

But when i tried to access Jenkins URL http://jenkins.company.com/jenkins on browser getting below error.

image

$ kubectl describe service jenkins-svc -n jenkins
Name:              jenkins-svc
Namespace:         jenkins
Labels:            app=jenkins
Annotations:       <none>
Selector:          app=jenkins
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.103.54.142
IPs:               10.103.54.142
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.2.2:8080
Session Affinity:  None
Events:            <none>

For example 16.20.21.6 jenkins.company.com entry already added in all hosts (HAproxy, Master & Worker) machines etc/hosts file.

Also in all host machine ping jenkins.company.com able to resolve my HA-Proxy IP.

Also i have validated this link https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ DNS resolution couldn't find problem.

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-57575c5f89-h2j74   1/1     Running   0          6d5h
coredns-57575c5f89-t5hbb   1/1     Running   0          6d5h

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191

$ kubectl get svc --namespace=kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   6d5h

Earlier shared output of kubectl exec -it -n nginx-ingress nginx-ingress-l7z86 -- nginx -T at the last line referring below.

proxy_pass http://jenkins-jenkins-ingress-jenkins.company.com-jenkins-svc-80;

Please let me proxy_pass has the correct syntax? Please let me know your views.

shaun-nx commented 1 year ago

Please let me proxy_pass has the correct syntax? Please let me know your views.

I overlooked this. In your Ingress deployment file you are using a community ingress annotation for rewrites

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # Community annotation
    nginx.ingress.kubernetes.io/rewrite-target: /$2

To add the path rewrite to your proxy_pass you will need to add this annotaiton:

  annotations:
    nginx.org/rewrites: "serviceName=jenkins-svc rewrite=/jenkins"

This will make your proxy_pass look like this:

proxy_pass http://jenkins-jenkins-ingress-jenkins.company.com-jenkins-svc-80/jenkins;

This will pass the /jenkins path to your upstream. Please let me know if this works.

mohanss08 commented 1 year ago

Hi Shaun,

Have updated my ingress-resource.yml file and applied. However still no success.

My current file as follows.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: jenkins
  annotations:
    # nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.org/rewrites: "serviceName=jenkins-svc rewrite=/jenkins"
spec:
  ingressClassName: nginx
  rules:
  - host: jenkins.company.com
    http:
      paths:
      - path: /jenkins
        pathType: Prefix
        backend:
          service:
            name: jenkins-svc
            port: 
              number: 80
shaun-nx commented 1 year ago

Hmmm, that's very strange...

Can you do two things for me?

  1. Confirm that the proxy_path value was updated to add /jenkins to the end

  2. Get the logs from the Nginx Ingress Controller pods This can be done with kubectl logs nginx-ingress-69z69 -n nginx-ingress (Note: try to access the login page first and then print the logs)

You might need to get the logs of all 3 of your pods. So run the same command for all 3 of them.

mohanss08 commented 1 year ago

Hi Shaun,

  1. Proxy_path updated like as follows proxy_pass http://jenkins-jenkins-ingress-jenkins.company.com-jenkins-svc-80/jenkins;
  2. Nginx Ingress Controller pods(All three pods) logs as follows.
kubectl logs pod/nginx-ingress-69z69 -n nginx-ingress
2023/01/17 05:52:43 [notice] 23#23: signal 29 (SIGIO) received
2023/01/17 05:52:43 [notice] 23#23: signal 17 (SIGCHLD) received from 157
2023/01/17 05:52:43 [notice] 23#23: signal 17 (SIGCHLD) received from 168
2023/01/17 05:52:43 [notice] 23#23: worker process 140 exited with code 0
2023/01/17 05:52:43 [notice] 23#23: worker process 168 exited with code 0
2023/01/17 05:52:43 [notice] 23#23: signal 29 (SIGIO) received

kubectl logs pod/nginx-ingress-l7z86 -n nginx-ingress
2023/01/17 05:52:43 [notice] 29#29: signal 29 (SIGIO) received
2023/01/17 05:52:43 [notice] 29#29: signal 17 (SIGCHLD) received from 119
2023/01/17 05:52:43 [notice] 29#29: worker process 126 exited with code 0
2023/01/17 05:52:43 [notice] 29#29: worker process 160 exited with code 0
2023/01/17 05:52:43 [notice] 29#29: worker process 165 exited with code 0
2023/01/17 05:52:43 [notice] 29#29: signal 29 (SIGIO) received

kubectl logs pod/nginx-ingress-wsjfv -n nginx-ingress
2023/01/17 05:52:43 [notice] 31#31: signal 17 (SIGCHLD) received from 155
2023/01/17 05:52:43 [notice] 31#31: worker process 155 exited with code 0
2023/01/17 05:52:43 [notice] 31#31: signal 29 (SIGIO) received
2023/01/17 05:52:43 [notice] 31#31: signal 17 (SIGCHLD) received from 125
2023/01/17 05:52:43 [notice] 31#31: worker process 125 exited with code 0
2023/01/17 05:52:43 [notice] 31#31: signal 29 (SIGIO) received

On the browser when i tried to access my jenkins URL http://jenkins.company.com/jenkins nginx ingress pods it printed above logs.

But pods are running fine.

$ kubectl get pods --namespace=nginx-ingress
NAME                  READY   STATUS    RESTARTS   AGE
nginx-ingress-69z69   1/1     Running   0          17m
nginx-ingress-l7z86   1/1     Running   0          17m
nginx-ingress-wsjfv   1/1     Running   0          17m
$ kubectl describe ingress jenkins-ingress -n jenkins
Name:             jenkins-ingress
Labels:           <none>
Namespace:        jenkins
Address:
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host             Path  Backends
  ----             ----  --------
  jenkins.company.com
                   /jenkins   jenkins-svc:80 (10.244.2.2:8080)
Annotations:       nginx.org/rewrites: serviceName=jenkins-svc rewrite=/jenkins
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  15s   nginx-ingress-controller  Configuration for jenkins/jenkins-ingress was added or updated
  Normal  AddedOrUpdated  15s   nginx-ingress-controller  Configuration for jenkins/jenkins-ingress was added or updated
  Normal  AddedOrUpdated  15s   nginx-ingress-controller  Configuration for jenkins/jenkins-ingress was added or updated
shaun-nx commented 1 year ago

Hi Mohan.

Based on the logs it doesn't look like the Ingress Controller has even processed the request. There should be a log line similar to this one:

127.0.0.1 - - [16/Jan/2023:13:54:53 +0000] "GET /jenkins HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" "-"

Check to see if you had any 404 log from the Ingress Controller pod.

There may be information getting lost from the request sent from the HAProxy server and the Ingress Controller. For example, the path /jenkins might not get sent over. I would reach out to them on their github page in case something is misconfigured with HAProxy: https://github.com/haproxy/haproxy

In the mean time, can you confirm that you are able to access the Jenkins login page locally from within your Kubernetes cluster?

First, port-forward port 80 of the Ingress Controller pod. This can be any one of the 3 pods. I would scale the deployment down to just 1 Ingress Controller pod while you are testing:

kubectl port-forward nginx-ingress-69z69 -n nginx-ingress 8080:80

Then, run this curl command:

curl -v --resolve jenkins.company.com:8080:127.0.0.1 http://jenkins.company.com:8080/jenkins/ --insecure

NOTE: I get a 403 as it is redirecting me to the initial Jenkins setup page before I can login (same as the original screenshot I posted):

* Added jenkins.company.com:8080:127.0.0.1 to DNS cache
* Hostname jenkins.company.com was found in DNS cache
*   Trying 127.0.0.1:8080...
* Connected to jenkins.company.com (127.0.0.1) port 8080 (#0)
> GET /jenkins/ HTTP/1.1
> Host: jenkins.company.com:8080
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Server: nginx/1.23.3
< Date: Tue, 17 Jan 2023 11:20:09 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 611
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< Set-Cookie: JSESSIONID.cdcab5c2=node01dicyv8vp1781xd1nujekjuy15.node0; Path=/jenkins; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Hudson: 1.395
< X-Jenkins: 2.375.2
< X-Jenkins-Session: 0dd17510
<
<html><head><meta http-equiv='refresh' content='1;url=/jenkins/login?from=%2Fjenkins%2Fjenkinsjenkins%2F'/><script>window.location.replace('/jenkins/login?from=%2Fjenkins%2Fjenkinsjenkins%2F');</script></head><body style='background-color:white; color:white;'>

Authentication required
<!--
-->

* Connection #0 to host jenkins.company.com left intact
</body></html>

This is the log line I get from the Ingress Controller pod:

127.0.0.1 - - [17/Jan/2023:11:20:09 +0000] "GET /jenkins/ HTTP/1.1" 403 611 "-" "curl/7.79.1" "-"

Now, if I try to access the page from my browser (this is still from within the k8s cluster and not HAProxy) I get the setup page: image

In this case I get this log line with a 200 code

127.0.0.1 - - [17/Jan/2023:11:28:25 +0000] "GET /jenkins/login?from=%2Fjenkins%2F HTTP/1.1" 200 1603 "http://jenkins.company.com:8080/jenkins" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" "-"

There is an example here for using rewrite with the Ingress resource that may be helpful as well: https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/ingress-resources/rewrites

jasonwilliams14 commented 1 year ago

@mohanss08 Just a couple clarification question. Are you trying to rewrite in the browser for the URL? Looking at your resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: jenkins
  annotations:
    # nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.org/rewrites: "serviceName=jenkins-svc rewrite=/jenkins"
spec:
  ingressClassName: nginx
  rules:
  - host: jenkins.company.com
    http:
      paths:
      - path: /jenkins
        pathType: Prefix
        backend:
          service:
            name: jenkins-svc
            port: 
              number: 80

The path configured for NGINX Ingress is: jenkins.company.com/jenkins

However, you have setup a rewrite rule for that exact path. So, are you looking or need to do a rewrite? or do you just need to access jenkins.company.com/jenkins

mohanss08 commented 1 year ago

Thanks for the information. I shall check and come back soon as possible. Thank you!

mohanss08 commented 1 year ago

Hi Shaun,

I don't see 404 related errors in Ingress controller pod.

On the cluster host machine executed below commands and for curl i can see the similar output that you shared above.

kubectl port-forward nginx-ingress-69z69 -n nginx-ingress 8080:80

curl -v --resolve jenkins.company.com:8080:127.0.0.1 http://jenkins.compay.com:8080/jenkins/ --insecure

Also below log lines able to see from Ingress controller pod.

127.0.0.1 - - [20/Jan/2023:03:31:17 +0000] "GET /jenkins/ HTTP/1.1" 403 577 "-" "curl/7.58.0" "-"

Next on my cluster host browser able to access http://jenkins.company.com:8080/jenkins

Also able to access jenkins URL outside of my cluster system by adding my cluster node IP with jenkins.company.com in hosts file.

So this confirms that my HAProxy not forwarding the request to my cluster nginx-ingress controller pod? The problem with my HAProxy side is that right?

shaun-nx commented 1 year ago

Hi @mohanss08 I read through the issue you opened on the haproxy repo. I'm glad to hear that you were able to resolve the issue 😄 I'll close this issue on our end. Thanks for taking the time to reach out to us!

shaun-nx commented 1 year ago

Resolved by #2007

mohanss08 commented 1 year ago

Hi Shaun, Yes it was resolved and thanks a lot for your kind support 😀. Appreciate your help on this