Canary ingress does not receive any traffic

[dintel]

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): v0.44.0

Kubernetes version (use kubectl version): v1.20.7

Environment:

• Cloud provider or hardware configuration: AWS

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools: EKS provsioned using Terraform

• Basic cluster related info:

  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:21:44Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609", GitCommit:"d886092805d5cc3a47ed5cf0c43de38ce442dfcb", GitTreeState:"clean", BuildDate:"2021-07-31T00:29:12Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

• How was the ingress-nginx-controller installed: helm template

• Current State of the controller: Deployed and running

• Current state of ingress object, if applicable: Deployed, but canary is misbehaving.

• Others:

What happened:

After deploying ingress and canary ingress, canary is not getting any traffic, although it's weihgt is 25.

What you expected to happen:

Canary ingress should have received traffic.

How to reproduce it:

Install ingress-nginx v0.44.0 using Helm on Kubernetes cluster. Create 3 ingress objects web, web2 and web-rc using following manifests (order matters):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: default
spec:
  rules:
  - host: web.devel
    http:
      paths:
      - backend:
          service:
            name: web
            port:
              number: 443
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - web.example.com

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web2
  namespace: default
spec:
  rules:
  - host: web2.devel
    http:
      paths:
      - backend:
          service:
            name: web-rc
            port:
              number: 443
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - web.example.com

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: true
    nginx.ingress.kubernetes.io/canary-weight: 25
  name: web-rc
  namespace: default
spec:
  rules:
  - host: web.devel
    http:
      paths:
      - backend:
          service:
            name: web-rc
            port:
              number: 443
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - web.example.com

You can ensure that web-rc service will not get any traffic by checking ingress-nginx configuration using command kubectl ingress-nginx backends -n ingress-nginx --backend default-web-rc-443 that shows weight of web-rc is 0.

Anything else we need to know:

This is most likely because web-rc service is used by web2 ingress and internally ingress-nginx defines single backend for each Kubernetes service.

/kind bug

[k8s-ci-robot]

@dintel: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

/remove-kind bug /kind support Last time I checked canary was working as expected. If you can write a step by step procedure, that includes commands and yaml files, to use on a minikube or kind cluster, it will help someone to try to reproduce the problem and make progress.

At the very least, you can show 50-100 http requests being made and then show corresponding logs and statistics that display the problem that all requests went to the main pod.

someone may try to check/reproduce aync, but your step-by-step instructions will help a lot.

[dintel]

Reproduction steps:

1. Install minikube and enable ingress-nginx (minikube addons enable ingress).
2. Create web and web-rc deployments:
   1. web - kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
   2. web-rc - kubectl create deployment web-rc --image=gcr.io/google-samples/hello-app:2.0
3. Create web and web-rc services.
   1. web - kubectl expose deployment web --port=8080
   2. web-rc - kubectl expose deployment web-rc --port=8080
4. Create web, web2 and web-rc ingress objects as described below (order matters!).
   1. web

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
      name: web
      spec:
      rules:
      - host: web.local
       http:
         paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: web
                 port:
                   number: 8080

   2. web2

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
      name: web2
      spec:
      rules:
      - host: web2.local
       http:
         paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: web-rc
                 port:
                   number: 8080

   3. web-rc

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
      name: web-rc
      annotations:
      nginx.ingress.kubernetes.io/canary: "true"
      nginx.ingress.kubernetes.io/canary-weight: "25"
      spec:
      rules:
      - host: web.local
       http:
         paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: web-rc
                 port:
                   number: 8080

5. Validate that ingress-nginx backend is configured incorrectly for service web-rc by running kubectl ingress-nginx backends -n ingress-nginx --backend default-web-rc-8080. Verify that trafficShapingPolicy.weight=0 in the JSON returned.
6. Add web.local web web2.local to your /etc/hosts file - run sudo sh -c 'echo $(minikube ip) web.local web2.local >> /etc/hosts'
7. Validate that all requests to web.local are reaching web pods only by running for i in $(seq 20); do curl web.local; done

On step 5 you can already see what the problem is. It happens because when canary ingress is created there is already another non-canary ingress (web2) using same service as canary ingress. It's not that all of canary functionality is broken, but rather a specific case in which canary does not work.

[longwuyuan]

Thank you very much for the reproduce procedure. This procedure is not the best procedure for making progress.

If you can help, we can make some more progress. If you can repeat the test with below changes, it will help.

• Please change step2 and remove the rewrite-targetannotation from all the ingresses ;

  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
  name: web
  spec:
  rules:
  - host: web.local
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: web
              port:
                number: 8080
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
  name: web2
  spec:
  rules:
  - host: web2.local
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: web-rc
              port:
                number: 8080
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
  name: web-rc
  annotations:
  nginx.ingress.kubernetes.io/canary: "true"
  nginx.ingress.kubernetes.io/canary-weight: "25"
  spec:
  rules:
  - host: web.local
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: web-rc
              port:
                number: 8080

• Please change step3 and create web and web-rc services as --type ClusterIP instead of --type NodePort. If you use expose command and do not specify service-type, then the Default is type ClusterIP.

    kubectl expose deployment web-rc  --port=8080
    kubectl expose deployment web-rc  --port=8080

[strongjz]

/assign @longwuyuan /priority important-longterm

[dintel]

@longwuyuan I just reran reproduction with your changes and the result is the same. Output of kubectl ingress-nginx backends -n ingress-nginx --backend default-web-rc-8080 shows:

...
  "trafficShapingPolicy": {
    "cookie": "",
    "header": "",
    "headerPattern": "",
    "headerValue": "",
    "weight": 0
  },
...

Notice the weight is 0, while it's expected to be 25. From what I understand this happens because internally ingress-nginx defines backend for each service that is used by any ingress. Since web-rc service is used by both web2 and web-rc ingresses and web2 is not canary, therefore weight=0. That's why order of ingress creation matters. If web-rc ingress is created before web2 ingress canary will work, but then web2 ingress will be broken.

P.S. @longwuyuan can I add your changes to reproduction instruction in previous comment?

[dintel]

@longwuyuan any update?

[longwuyuan]

hi, Sorry for delay. Can you kindly show ;

• kubectl get svc -A -o wide | grep -i ingress
• kubectl get ing -A -o wide
• kubectl -n describe ing # for your app
• kubectl -n describe svc # for your app
• Your complete curl command
• Your logs of the controller pod for your curl command

[dintel]

Hi @longwuyuan Following are command outputs and logs as you requested.

$ kubectl get svc -A -o wide | grep -i ingress
ingress-nginx   ingress-nginx-controller             NodePort    10.100.84.27     <none>        80:31639/TCP,443:31694/TCP   3m21s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.107.206.226   <none>        443/TCP                      3m21s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

$ kubectl get ing -A -o wide
NAMESPACE   NAME     CLASS   HOSTS        ADDRESS     PORTS   AGE
default     web      nginx   web.local    localhost   80      2m32s
default     web-rc   nginx   web.local    localhost   80      117s
default     web2     nginx   web2.local   localhost   80      2m32s

$ kubectl describe ing web
Name:             web
Namespace:        default
Address:          localhost
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  web.local   
              /   web:8080 (172.17.0.4:8080)
Annotations:  <none>
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    2m24s (x2 over 2m51s)  nginx-ingress-controller  Scheduled for sync

$ kubectl describe svc web
Name:              web
Namespace:         default
Labels:            app=web
Annotations:       <none>
Selector:          app=web
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.101.61.248
IPs:               10.101.61.248
Port:              <unset>  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.4:8080
Session Affinity:  None
Events:            <none>

curl command: for i in $(seq 20); do curl web.local; done ingress-nginx logs

ingress-nginx-controller-5f66978484-wxj8j controller -------------------------------------------------------------------------------
ingress-nginx-controller-5f66978484-wxj8j controller NGINX Ingress controller
ingress-nginx-controller-5f66978484-wxj8j controller   Release:       v1.0.4
ingress-nginx-controller-5f66978484-wxj8j controller   Build:         9b78b6c197b48116243922170875af4aa752ee59
ingress-nginx-controller-5f66978484-wxj8j controller   Repository:    https://github.com/kubernetes/ingress-nginx
ingress-nginx-controller-5f66978484-wxj8j controller   nginx version: nginx/1.19.9
ingress-nginx-controller-5f66978484-wxj8j controller 
ingress-nginx-controller-5f66978484-wxj8j controller -------------------------------------------------------------------------------
ingress-nginx-controller-5f66978484-wxj8j controller 
ingress-nginx-controller-5f66978484-wxj8j controller W1202 21:27:20.767086      11 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.767236      11 main.go:221] "Creating API client" host="https://10.96.0.1:443"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.784828      11 main.go:265] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.3" state="clean" commit="c92036820499fedefec0f847e2054d824aea6cd1" platform="linux/amd64"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.864325      11 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.879684      11 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.908150      11 nginx.go:253] "Starting NGINX Ingress controller"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.915322      11 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"70360134-58e2-4efb-a361-6cd0d2b441fe", APIVersion:"v1", ResourceVersion:"489", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.915386      11 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"fcdd694e-7c7f-4589-98df-26b4881cecbc", APIVersion:"v1", ResourceVersion:"490", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:20.915399      11 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"2b80b772-b76f-4b41-8b20-2dbeb7366ce2", APIVersion:"v1", ResourceVersion:"487", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.110589      11 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.110685      11 nginx.go:295] "Starting NGINX process"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.112045      11 nginx.go:315] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.112190      11 controller.go:152] "Configuration changes detected, backend reload required"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.123106      11 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.123394      11 status.go:84] "New leader elected" identity="ingress-nginx-controller-5f66978484-wxj8j"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.172062      11 controller.go:169] "Backend successfully reloaded"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.172743      11 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5f66978484-wxj8j", UID:"869b353c-f0f2-482e-bf06-0e6cd0160ad3", APIVersion:"v1", ResourceVersion:"524", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:22.172799      11 controller.go:180] "Initial sync, sleeping for 1 second"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.107797      11 main.go:101] "successfully validated configuration, accepting" ingress="web/default"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.114231      11 store.go:371] "Found valid IngressClass" ingress="default/web" ingressclass="nginx"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.114793      11 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web", UID:"c9d040a6-91eb-4f14-8274-333f1d82fa03", APIVersion:"networking.k8s.io/v1", ResourceVersion:"673", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.116469      11 controller.go:152] "Configuration changes detected, backend reload required"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.188162      11 main.go:101] "successfully validated configuration, accepting" ingress="web2/default"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.191398      11 store.go:371] "Found valid IngressClass" ingress="default/web2" ingressclass="nginx"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.191870      11 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web2", UID:"3bd7ab28-4565-4afc-8cae-15b6e45aa72d", APIVersion:"networking.k8s.io/v1", ResourceVersion:"675", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.208083      11 controller.go:169] "Backend successfully reloaded"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:55.208635      11 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5f66978484-wxj8j", UID:"869b353c-f0f2-482e-bf06-0e6cd0160ad3", APIVersion:"v1", ResourceVersion:"524", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:58.450033      11 controller.go:152] "Configuration changes detected, backend reload required"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:58.521943      11 controller.go:169] "Backend successfully reloaded"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:27:58.527862      11 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5f66978484-wxj8j", UID:"869b353c-f0f2-482e-bf06-0e6cd0160ad3", APIVersion:"v1", ResourceVersion:"524", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:22.138058      11 status.go:300] "updating Ingress status" namespace="default" ingress="web" currentValue=[] newValue=[{IP: Hostname:localhost Ports:[]}]
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:22.138376      11 status.go:300] "updating Ingress status" namespace="default" ingress="web2" currentValue=[] newValue=[{IP: Hostname:localhost Ports:[]}]
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:22.149461      11 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web", UID:"c9d040a6-91eb-4f14-8274-333f1d82fa03", APIVersion:"networking.k8s.io/v1", ResourceVersion:"701", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:22.153223      11 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web2", UID:"3bd7ab28-4565-4afc-8cae-15b6e45aa72d", APIVersion:"networking.k8s.io/v1", ResourceVersion:"702", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:30.646012      11 main.go:101] "successfully validated configuration, accepting" ingress="web-rc/default"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:30.653152      11 store.go:371] "Found valid IngressClass" ingress="default/web-rc" ingressclass="nginx"
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:28:30.655178      11 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web-rc", UID:"56df0260-770e-45a9-b12a-9662d9730694", APIVersion:"networking.k8s.io/v1", ResourceVersion:"713", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:29:22.137222      11 status.go:300] "updating Ingress status" namespace="default" ingress="web-rc" currentValue=[] newValue=[{IP: Hostname:localhost Ports:[]}]
ingress-nginx-controller-5f66978484-wxj8j controller I1202 21:29:22.144587      11 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web-rc", UID:"56df0260-770e-45a9-b12a-9662d9730694", APIVersion:"networking.k8s.io/v1", ResourceVersion:"757", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:29:47 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.002 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 f68d3da477ca736d13dd5dc617b056c1
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:29:53 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 d35bca65f19a06feec8fcb07955caa64
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:29:58 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 6088f32defc14cd96f323a323229eabe
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:03 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 6650bb0f68cf69866c5eac87b45a3f85
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:08 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 4d28491a703c7e39cffe918854661f3f
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:19 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 41effd7fc03e025b55e61efc174a775a
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:24 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 3e990fc7f556ec7487ff631f2e3256d1
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:29 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 bb22e60a3dbe8924214dea184c3fc5d4
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:34 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 3e38ece4bf0a60975ae3ea8a420d8091
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:39 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 7470545096de11bbc49d5fd68761d903
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:44 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 7f837983821d4e742e1ac0374f8cd11d
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:50 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 3e8a2b3b9502d637c3fbbac3e459162a
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:30:55 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 8775b9c642dc4253ef1c40a2b7f02df7
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:00 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 60b1d85c28050d8b3e3d70f22f65a020
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:05 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 fea821d8d03a9971878bd7f63408927b
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:10 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 1d07a7c121c9d0c2c265baa4a32366c3
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:13 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 c9e136ed4d9dd8b4dbb97e3296e3d109
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:15 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 154e9ac24ee7e33ada1ad9c734ff00d3
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:20 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 556cc48c2a25f4e476f7cc5c2b2f284e
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:25 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 dc7002dcdd4f3fae2ff39d92067a1373
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:31 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 b46a91c008031487e56ecdb2cdc88294
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:36 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 16affd6c84c671003b1330b72905cab6
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:41 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 e789d86ab4b64920588b3e8267edc560
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:46 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 a1d9dd70c6ef8368d4bf3b91b939a9ee
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:51 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 50d32fdc5f3fdc5475540dda4cb627a1
ingress-nginx-controller-5f66978484-wxj8j controller 192.168.64.1 - - [02/Dec/2021:21:31:56 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 01005de514061bdb3d0f7c5f3853f361

[longwuyuan]

Sorry if my earlier list of commands was not explicit. I thought the intention was obvious to you. The info you provided is not complete. I don't see the describe output for all the ingresses and all the services.

[dintel]

Hi @longwuyuan Here is updated information:

$ kubectl get svc -A -o wide | grep -i ingress
ingress-nginx   ingress-nginx-controller             NodePort    10.106.133.161   <none>        80:31503/TCP,443:32598/TCP   4m54s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.101.68.65     <none>        443/TCP                      4m54s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

$ kubectl get ing -A -o wide
NAMESPACE   NAME     CLASS   HOSTS        ADDRESS     PORTS   AGE
default     web      nginx   web.local    localhost   80      3m59s
default     web-rc   nginx   web.local    localhost   80      3m27s
default     web2     nginx   web2.local   localhost   80      3m50s

$ kubectl describe svc web
Name:              web
Namespace:         default
Labels:            app=web
Annotations:       <none>
Selector:          app=web
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.104.207.68
IPs:               10.104.207.68
Port:              <unset>  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.4:8080
Session Affinity:  None
Events:            <none>

$ kubectl describe svc web-rc 
Name:              web-rc
Namespace:         default
Labels:            app=web-rc
Annotations:       <none>
Selector:          app=web-rc
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.105.20.174
IPs:               10.105.20.174
Port:              <unset>  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.5:8080
Session Affinity:  None
Events:            <none>

$ kubectl describe ing web
Name:             web
Namespace:        default
Address:          localhost
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  web.local   
              /   web:8080 (172.17.0.4:8080)
Annotations:  <none>
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    5m13s (x2 over 5m32s)  nginx-ingress-controller  Scheduled for sync

$ kubectl describe ing web2
Name:             web2
Namespace:        default
Address:          localhost
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  web2.local  
              /   web-rc:8080 (172.17.0.5:8080)
Annotations:  <none>
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    5m15s (x2 over 5m25s)  nginx-ingress-controller  Scheduled for sync

$ kubectl describe ing web-rc
Name:             web-rc
Namespace:        default
Address:          localhost
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  web.local   
              /   web-rc:8080 (172.17.0.5:8080)
Annotations:  nginx.ingress.kubernetes.io/canary: true
              nginx.ingress.kubernetes.io/canary-weight: 25
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    4m17s (x2 over 5m4s)  nginx-ingress-controller  Scheduled for sync

curl command: for i in $(seq 20); do curl web.local; done ingress-nginx logs

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.4
  Build:         9b78b6c197b48116243922170875af4aa752ee59
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

W1219 12:22:23.062430       7 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1219 12:22:23.062778       7 main.go:221] "Creating API client" host="https://10.96.0.1:443"
I1219 12:22:23.082598       7 main.go:265] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.3" state="clean" commit="c92036820499fedefec0f847e2054d824aea6cd1" platform="linux/amd64"
I1219 12:22:23.358774       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1219 12:22:23.375894       7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1219 12:22:23.388153       7 nginx.go:253] "Starting NGINX Ingress controller"
I1219 12:22:23.401688       7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"5d3b3e4b-86c7-4ca3-9c87-957a60f20a9f", APIVersion:"v1", ResourceVersion:"485", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1219 12:22:23.407099       7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"8fa3129b-9a64-403b-978a-3c9b0c79055c", APIVersion:"v1", ResourceVersion:"488", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1219 12:22:23.407139       7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"78554b5c-3dff-4455-8499-76fd1a5259c4", APIVersion:"v1", ResourceVersion:"489", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1219 12:22:24.590148       7 nginx.go:295] "Starting NGINX process"
I1219 12:22:24.590216       7 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I1219 12:22:24.590462       7 nginx.go:315] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1219 12:22:24.590643       7 controller.go:152] "Configuration changes detected, backend reload required"
I1219 12:22:24.599267       7 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader
I1219 12:22:24.599972       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-5f66978484-gxx2g"
I1219 12:22:24.645883       7 controller.go:169] "Backend successfully reloaded"
I1219 12:22:24.646119       7 controller.go:180] "Initial sync, sleeping for 1 second"
I1219 12:22:24.646166       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5f66978484-gxx2g", UID:"a4562fd7-1762-4226-8d55-3cedc4197ab0", APIVersion:"v1", ResourceVersion:"586", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1219 12:23:05.271467       7 main.go:101] "successfully validated configuration, accepting" ingress="web/default"
I1219 12:23:05.275775       7 store.go:371] "Found valid IngressClass" ingress="default/web" ingressclass="nginx"
I1219 12:23:05.276232       7 controller.go:152] "Configuration changes detected, backend reload required"
I1219 12:23:05.278932       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web", UID:"38679b57-294d-4b47-b31e-309be57b88fb", APIVersion:"networking.k8s.io/v1", ResourceVersion:"687", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 12:23:05.323633       7 controller.go:169] "Backend successfully reloaded"
I1219 12:23:05.324878       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5f66978484-gxx2g", UID:"a4562fd7-1762-4226-8d55-3cedc4197ab0", APIVersion:"v1", ResourceVersion:"586", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1219 12:23:14.258452       7 main.go:101] "successfully validated configuration, accepting" ingress="web2/default"
I1219 12:23:14.263168       7 store.go:371] "Found valid IngressClass" ingress="default/web2" ingressclass="nginx"
I1219 12:23:14.263458       7 controller.go:152] "Configuration changes detected, backend reload required"
I1219 12:23:14.263513       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web2", UID:"e27e0afb-0085-4109-8a56-a64ea4d56fca", APIVersion:"networking.k8s.io/v1", ResourceVersion:"696", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 12:23:14.329763       7 controller.go:169] "Backend successfully reloaded"
I1219 12:23:14.330249       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5f66978484-gxx2g", UID:"a4562fd7-1762-4226-8d55-3cedc4197ab0", APIVersion:"v1", ResourceVersion:"586", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1219 12:23:24.607674       7 status.go:300] "updating Ingress status" namespace="default" ingress="web2" currentValue=[] newValue=[{IP: Hostname:localhost Ports:[]}]
I1219 12:23:24.608269       7 status.go:300] "updating Ingress status" namespace="default" ingress="web" currentValue=[] newValue=[{IP: Hostname:localhost Ports:[]}]
I1219 12:23:24.614712       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web", UID:"38679b57-294d-4b47-b31e-309be57b88fb", APIVersion:"networking.k8s.io/v1", ResourceVersion:"709", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 12:23:24.617327       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web2", UID:"e27e0afb-0085-4109-8a56-a64ea4d56fca", APIVersion:"networking.k8s.io/v1", ResourceVersion:"710", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 12:23:37.243366       7 main.go:101] "successfully validated configuration, accepting" ingress="web-rc/default"
I1219 12:23:37.246858       7 store.go:371] "Found valid IngressClass" ingress="default/web-rc" ingressclass="nginx"
I1219 12:23:37.247565       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web-rc", UID:"25b94186-6a11-425d-9978-7de06c2df02b", APIVersion:"networking.k8s.io/v1", ResourceVersion:"723", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1219 12:24:24.607419       7 status.go:300] "updating Ingress status" namespace="default" ingress="web-rc" currentValue=[] newValue=[{IP: Hostname:localhost Ports:[]}]
I1219 12:24:24.614572       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"web-rc", UID:"25b94186-6a11-425d-9978-7de06c2df02b", APIVersion:"networking.k8s.io/v1", ResourceVersion:"765", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
192.168.64.1 - - [19/Dec/2021:12:36:13 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 aa85c0a0a467e008279c11b4658fba69
192.168.64.1 - - [19/Dec/2021:12:36:18 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 b016437d11453d2673e505baa870d5d3
192.168.64.1 - - [19/Dec/2021:12:36:24 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 bfd9d5001125d829760e8d829016289e
192.168.64.1 - - [19/Dec/2021:12:36:29 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 32f72f4428071afba195aeb77bbddd12
192.168.64.1 - - [19/Dec/2021:12:36:34 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 1229e31b7acfd170cb42659fedf29d83
192.168.64.1 - - [19/Dec/2021:12:36:39 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 da6bf648a962c9868e8bff864780fc27
192.168.64.1 - - [19/Dec/2021:12:36:44 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 23cf51012791f41aaad869d573bd7a03
192.168.64.1 - - [19/Dec/2021:12:36:49 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.002 [default-web-8080] [] 172.17.0.4:8080 60 0.002 200 7299434165de688afc1eed7fd0bba79b
192.168.64.1 - - [19/Dec/2021:12:36:54 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 6c5d051a3c735590791acb2af78adb3e
192.168.64.1 - - [19/Dec/2021:12:37:00 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 ca8da3d8076c4ab16aa610885b23f2d4
192.168.64.1 - - [19/Dec/2021:12:37:05 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 306988618ebbefe8d422370debbb1697
192.168.64.1 - - [19/Dec/2021:12:37:10 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 0cb7a589927e3a5715d0450d03ca8659
192.168.64.1 - - [19/Dec/2021:12:37:15 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 523745df426303577d16fcfd224b77c1
192.168.64.1 - - [19/Dec/2021:12:37:20 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 0f6b17c3d20e26b77042ed01e1baa211
192.168.64.1 - - [19/Dec/2021:12:37:25 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 70710b8ee4799ddc6b343accdfefc635
192.168.64.1 - - [19/Dec/2021:12:37:30 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 4b9490d9ee6e72f660e4429f65a56700
192.168.64.1 - - [19/Dec/2021:12:37:36 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.000 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 4611e261a32ccc004e7798d780e6bd8d
192.168.64.1 - - [19/Dec/2021:12:37:41 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 fcf824075cc52d719a008dcd27f811f2
192.168.64.1 - - [19/Dec/2021:12:37:46 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.000 200 8431e0cb77679e792cf982cc17c829f1
192.168.64.1 - - [19/Dec/2021:12:37:51 +0000] "GET / HTTP/1.1" 200 60 "-" "curl/7.64.1" 73 0.001 [default-web-8080] [] 172.17.0.4:8080 60 0.001 200 31db01bf01b6e2f2ba78079b90faff60

[dintel]

@longwuyuan Adding here another command output:

$ kubectl ingress-nginx backends -n ingress-nginx --backend default-web-rc-8080
{
  "endpoints": [
    {
      "address": "172.17.0.5",
      "port": "8080"
    }
  ],
  "name": "default-web-rc-8080",
  "noServer": false,
  "port": 8080,
  "service": {
    "metadata": {
      "creationTimestamp": null
    },
    "spec": {
      "clusterIP": "10.105.20.174",
      "clusterIPs": [
        "10.105.20.174"
      ],
      "internalTrafficPolicy": "Cluster",
      "ipFamilies": [
        "IPv4"
      ],
      "ipFamilyPolicy": "SingleStack",
      "ports": [
        {
          "port": 8080,
          "protocol": "TCP",
          "targetPort": 8080
        }
      ],
      "selector": {
        "app": "web-rc"
      },
      "sessionAffinity": "None",
      "type": "ClusterIP"
    },
    "status": {
      "loadBalancer": {}
    }
  },
  "sessionAffinityConfig": {
    "cookieSessionAffinity": {
      "name": ""
    },
    "mode": "",
    "name": ""
  },
  "sslPassthrough": false,
  "trafficShapingPolicy": {
    "cookie": "",
    "header": "",
    "headerPattern": "",
    "headerValue": "",
    "weight": 0
  },
  "upstreamHashByConfig": {
    "upstream-hash-by-subset-size": 3
  }
}

You can see here that trafficShapingPolicy is configured incorrectly with weight=0. My guess is that this is because each Kubernetes service is mapped internally by ingress-nginx to backend object. Since service web-rc is already used by ingress web2, it's not reconfigured. If I am correct, then this is a bug in design and I suggest to just document such behavior.

[longwuyuan]

I don't even understand the configuration you have provided. Makes no sense to me, in the context of Canary deployment.

Double check what you are doing. I found an example so better check if the example works first https://v2-1.docs.kubesphere.io/docs/quick-start/ingress-canary/

[dintel]

@longwuyuan Can you be more specific about what does not make sense to you? I provided exact instructions how this issue can be reproduced and extensive details from one of reproductions. Reproduction is 100%. Canary always does not work when setup as described. How can I help more with this case?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[longwuyuan]

Please reopen if this is still an issue. We are working on some documentation related to canary deployments /close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7896#issuecomment-1092466769): >Please reopen if this is still an issue. We are working on some documentation related to canary deployments >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[rthamrin]

/reopen

I do the same, my canary is not working, apply canary-weight given to me an error

kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "50"

<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: ingress
  annotations:
    # kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  ingressClassName: nginx
  rules:
  - host: webservice.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web2
            port:
              number: 80
      - path: /web2
        pathType: Prefix
        backend:
          service:
            name: web2
            port:
              number: 80
      - path: /web3
        pathType: Prefix
        backend:
          service:
            name: web3
            port:
              number: 80

I add another annotation, but it was ok.

[k8s-ci-robot]

@rthamrin: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7896#issuecomment-1093557416): >/reopen > >I do the same, my canary is not working, apply canary-weight given to me an error > >kubernetes.io/ingress.class: nginx >nginx.ingress.kubernetes.io/canary: "true" >nginx.ingress.kubernetes.io/canary-weight: "50" > >``` > > >

404 Not Found

>

---

nginx > > > >``` > >``` >apiVersion: networking.k8s.io/v1 >kind: Ingress >metadata: > name: nginx-ingress > namespace: ingress > annotations: > # kubernetes.io/ingress.class: nginx > nginx.ingress.kubernetes.io/canary: "true" > nginx.ingress.kubernetes.io/canary-weight: "50" >spec: > ingressClassName: nginx > rules: > - host: webservice.com > http: > paths: > - path: / > pathType: Prefix > backend: > service: > name: web2 > port: > number: 80 > - path: /web2 > pathType: Prefix > backend: > service: > name: web2 > port: > number: 80 > - path: /web3 > pathType: Prefix > backend: > service: > name: web3 > port: > number: 80 >``` >I add another annotation, but it was ok. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.