search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.26k stars 8.21k forks source link

namespace scope not working as expected #9662

Open yong-jie-gong opened 1 year ago

yong-jie-gong commented 1 year ago

What happened: 1) configure nginx ingress controller with specified ingress class as below

     /nginx-ingress-controller
      --watch-namespace=test1
      --annotations-prefix=ingress.kubernetes.io
      --ingress-class=test-local

2) create one one service account such as "nginx-ingress-controller-sa" whose don't have get/list/watch permission on cluster resource "networking.k8s.io/ingressclasses" 3) start nginx-ingress-controller with service account nginx-ingress-controller-sa 4) create one global ingress in the same namespace for other global ingress controller

apiVersion: networking.k8s.io/v1
kind: Ingress
  annotations:
    kubernetes.io/ingress.class: alb
  name: test-global
  namespace: test1

5) ingress object "test-global" will be caught by nginx-ingrss-controller and write into nginx configuration file by mistake, this new path rule cause conflict with other default local ingress path rule

but if I restart nginx-ingress-controller, "test-global" path rule will be removed from nginx configuration. it means nginx-ingress-controller behave different on runtime and fresh start.

What you expected to happen: s nginx-ingress-controller should work in consistent way on runtime and fresh start.

if ingress object doesn't belongs current nginx-ingress-controller, it should be not caught and write into nginx path rule

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): nginx-ingress-controller --version

NGINX Ingress controller Release: v1.5.1 Build: git-f966ccc43 Repository: github nginx version: nginx/1.21.6

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.7-eks-fb459a0", GitCommit:"c240013134c03a740781ffa1436ba2688b50b494", GitTreeState:"clean", BuildDate:"2022-10-24T20:40:13Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

Environment: NA

How to reproduce this issue:

Anything else we need to know:

longwuyuan commented 1 year ago

/retitle namespace scope not working as expected

namespace-scope is something to be looked at. After the stabilization work is completed, I think this will get some attention

/triage accepted

yong-jie-gong commented 1 year ago

@longwuyuan Raised one PR #9712 to fix this issue

longwuyuan commented 1 year ago

@yong-jie-gong I think the details of the problem you are hinting at, are not explained here. Also I made a comment on your PR.

Please help out here. Please write a step-by-step instructions procedure here in such a way that someone can create a default minikube cluster or kind cluster and then copy/paste from your instructions to reproduce the problem. This is because you have made several assumptions in your post that actually are a deep dive into the configuration of the controller, but none of the impacting design aspects are clearly explained. For example, you can look at the helm chart methijd of installing the controller and the key:value pairs related to that in the values.yaml . There is also the service-account object in the static manifest https://github.com/kubernetes/ingress-nginx/blob/4f74e03aad663c7d72edaa36d8e874f1ebe1aa2d/deploy/static/provider/cloud/deploy.yaml#L11 and related permissions . That requires a caveat and explaining for customized service-account and permissions for the custom service-account

github-actions[bot] commented 1 year ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

davidcurrie commented 9 months ago

@longwuyuan - I have hit this issue and have a simple recreate. I believe this is also the same issue described in #9636.

  1. Deploy the controller scoped to a single namespace: 
    helm install ingress-nginx ingress-nginx -n ingress-nginx --create-namespace --repo https://kubernetes.github.io/ingress-nginx \
--set controller.service.type=ClusterIP,controller.ingressClassResource.name=test,controller.scope.enabled=true,controller.admissionWebhooks.enabled=false,rbac.scope=true --skip-crds
  2. Create an Ingress resource with a mismatched ingress class name: 
    cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: ingress-nginx
spec:
ingressClassName: wrong
rules:
- host: www.example.com
  http:
    paths:
      - pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80
        path: /
EOF
  3. The controller logs indicate that the Ingress is being ignored: 
    I1213 22:14:30.827574       6 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="ingress-nginx/example" error="ingress does not contain a valid IngressClass"
  4. Make some update to the Ingress e.g. 
    cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: ingress-nginx
spec:
ingressClassName: wrong
rules:
- host: www.example.com
  http:
    paths:
      - pathType: Prefix
        backend:
          service:
            name: example-service-changed
            port:
              number: 80
        path: /
EOF
  5. The controller now attempts to process the Ingress resource even though the class name doesn't match: 
    I1213 22:15:20.974712       6 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"example", UID:"a7bad8f1-b8f9-4edb-8a1a-13f0f762f671", APIVersion:"networking.k8s.io/v1", ResourceVersion:"21415", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W1213 22:15:20.975048       6 controller.go:1108] Error obtaining Endpoints for Service "ingress-nginx/example-service-change": no object matching key "ingress-nginx/example-service-change" in local store
I1213 22:15:20.975186       6 controller.go:190] "Configuration changes detected, backend reload required"
I1213 22:15:21.051719       6 controller.go:210] "Backend successfully reloaded"

I see that the multi-ingress controller docs state:

But, when user has deployed with scope.enabled, then the ingress class resource field is not used.

It doesn't really make it clear what "not used" means, but I'd expect the behavior to be consistent. Either it should mean that the Ingress is never processed (unless the deprecated annotation is used), or it is always processed.

To give a bit more background on my scenario: I have a cluster-wide ingress controller that is used to provide external access to the cluster. For a specific application, I want to use the upstream-hash-by capability of nginx to provide consistent routing across the pods behind a service. To support this, the application deploys its own nginx controller that specifies a unique ingress class name and is scoped to the namespace in which the application runs. The RBAC permissions are also scoped to the namespace. The application defines an Ingress resource, which specifies the required ingress class name via annotation (as the controller doesn't not create an IngressClass).

This all works fine, except that the application also defines another Ingress resource in the same namespace that specifies the ingress class for the cluster-wide controller. As per the recreate above, when the resource is first created, everything is fine. It is ignored by the namespaced controller and the cluster-wide controller picks it up and processes it. Unfortunately, when the cluster-wide controller controller updates the status, that causes the namespaced controller to kick off a sync, even though the ingress class does not match. The resource then flip-flops between the two controllers.

I believe the fix in the linked PR is good. It would mean that, in the update scenario, it would still check the annotation on the resource and consequently ignore it.

longwuyuan commented 9 months ago

@davidcurrie Please explicitly state what the issue is for you. Are you trying to install a instance of the ingress-nginx controller that processes ingresses only from one namespace ?

davidcurrie commented 9 months ago

@longwuyuan - answers as follows:

Are you trying to install a instance of the ingress-nginx controller that processes ingresses only from one namespace ?

Yes. The complicating factors though are that a) there are other ingress resources in that same namespace that should be processed by a cluster-wide controller, and b) the namespaced controller should not have any cluster-wide permissions. In particular, the namespaced controller cannot read IngressClass.

I have to check what real world case there is for creating a service type ClusterIP

As I said above, the application has a requirement for sharding when one component is calling another. We want to use nginx's upstream-hash-by to provide the sharding. The problem exists whatever the service type but this is our motivation for having a second controller that is namespaced.

I have to check what are the default values of the flags you are using in the helm install command

The recreate above contains all of the flags necessary to recreate the problem.

In case your final goal is to have multiple controller instances in the same cluster then try this

This works because it does not set rbac.scope=true. This means that it can read IngressClass and, as a consequence, a call to GetIngressClass is made on the update at https://github.com/kubernetes/ingress-nginx/blob/6ff70f015dd8b12d5e96adc6ef3c16e1c9060944/internal/ingress/controller/store/store.go#L475-L478. What the PR is suggesting is that the update should always call GetIngressClass because, even if it can't read IngressClass resources (which that function handles anyway) it may still be able to usefully process annotations.

longwuyuan commented 9 months ago
davidcurrie commented 9 months ago

@longwuyuan, thanks for providing some background. As previously stated, we're using this nginx instance for internal traffic within the application. As such, we deploy the controller as part of what is essentially a packaged application. To date, we have avoided creating cluster-scoped resources, making it easier to persuade administrators to deploy the application in a multi-tenant cluster.

I appreciate that this is very definitely an edge case. What bothers me most here is the inconsistency in behavior. If the controller does not have permission to list IngressClass, then on the addition of an Ingress resource it currently uses the information that it can access (i.e. the kubernetes.io/ingress.class annotation) and otherwise ignores resources, but on an update it processes all resources regardless.

On both an add and an update, I would argue that it should ignore Ingress resources that specify annotations that do not match the class name specified on the controller. What should happen if there is no annotation could go either way. It could ignore the resource as it does on add today, or it could process the resource as it does on update. Either way, they should both do the same thing!

Note that none of the above is really about namespacing, it's just about what happens when it can't list IngressClass.

kfox1111 commented 9 months ago

We've run into the same behavior as we run an ingress controller per tenant namespace to restrict the blast damage of any configuration issues (lots, saved us from them many times).

longwuyuan commented 9 months ago

@davidcurri tht udpate is helpful. I think that a ton of info needs to be exchanged for triaging what you explained. I will try to list in not any particular order

davidcurrie commented 9 months ago

@longwuyuan - I'm only describing what we are doing as motivation as to why this matters to us. In terms of reproduction, please just follow the steps in https://github.com/kubernetes/ingress-nginx/issues/9662#issuecomment-1854818281 (the configuration we package in our application is based on a helm template from this command).

I believe what I'm describing is exactly the same as in the issue description. The key points are that the controller does not have permission to list IngressClass and that there is then a difference in behavior when an add is detected (annotation respected) vs an update (annotation is ignored). If the controller does have permission to list IngressClass (which is what you get if you follow the instructions in the docs) then the annotation is respected in both cases.

longwuyuan commented 9 months ago

Thank you @davidcurrie . I need to go backwards towards basics as I am not able to understand the problem that needs to be solved in the code. This is how I understand the related current feature of the controller in v1.9.4 ;

If I am reading the info here right, I have to guess that the description of the problem to be solved is not detailed enough

image

The timestamps are different for the create as well as the edit events in the screenshot

Previously, in the ingress configuration, I had set the backend service name to the actually existing service "test0". Now I edited the ingress object yet again and changed the backend service name to a non-existing service called "test1". And even this was ignored by the controller

image

Hence I think that in my test, I saw the controller ignore a ingress correctly, as expected.

Since this is not working for you, the issue description needs to be more detailed for example, output of below commands in one single post ;

But this is all my opinion. You may just wait for comments from others on this issue and the related PR

davidcurrie commented 8 months ago

@longwuyuan - additional information as requested...

helm install ingress-nginx ingress-nginx -n ingress-nginx --create-namespace --repo https://kubernetes.github.io/ingress-nginx --set controller.ingressClassResource.enabled=false,controller.scope.enabled=true,rbac.scope=true --skip-crds
NAME            NAMESPACE       REVISION    UPDATED                                 STATUS      CHART               APP VERSION
ingress-nginx   ingress-nginx   1           2023-12-20 16:48:37.243605 +0000 UTC    deployed    ingress-nginx-4.8.3 1.9.4
NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           33s
kube-system     deployment.apps/coredns                    1/1     1            1           46h

NAMESPACE       NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.96.0.1        <none>        443/TCP                      46h
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.99.188.74     <pending>     80:31884/TCP,443:31212/TCP   33s
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.100.210.231   <none>        443/TCP                      33s
kube-system     service/kube-dns                             ClusterIP      10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       46h

NAMESPACE       NAME                                            READY   STATUS    RESTARTS      AGE
ingress-nginx   pod/ingress-nginx-controller-699745fc87-sbh4s   1/1     Running   0             33s
kube-system     pod/coredns-565d847f94-lxbcb                    1/1     Running   0             46h
kube-system     pod/etcd-minikube                               1/1     Running   0             46h
kube-system     pod/kube-apiserver-minikube                     1/1     Running   0             46h
kube-system     pod/kube-controller-manager-minikube            1/1     Running   0             46h
kube-system     pod/kube-proxy-pkkzn                            1/1     Running   0             46h
kube-system     pod/kube-scheduler-minikube                     1/1     Running   0             46h
kube-system     pod/storage-provisioner                         1/1     Running   1 (46h ago)   46h
Name:             ingress-nginx-controller-699745fc87-sbh4s
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.49.2
Start Time:       Wed, 20 Dec 2023 16:48:42 +0000
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.9.4
                  helm.sh/chart=ingress-nginx-4.8.3
                  pod-template-hash=699745fc87
Annotations:      <none>
Status:           Running
IP:               10.244.0.28
IPs:
  IP:           10.244.0.28
Controlled By:  ReplicaSet/ingress-nginx-controller-699745fc87
Containers:
  controller:
    Container ID:  docker://fa3454d538253237ae135cf60386432e8b73071427889db1ec457db2df5fb940
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --watch-namespace=$(POD_NAMESPACE)
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Wed, 20 Dec 2023 16:48:42 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-699745fc87-sbh4s (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bn7jd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-bn7jd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  60s   default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-699745fc87-sbh4s to minikube
  Normal  Pulled     60s   kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
  Normal  Created    60s   kubelet                   Created container controller
  Normal  Started    60s   kubelet                   Started container controller
  Normal  RELOAD     58s   nginx-ingress-controller  NGINX reload triggered due to a change in configuration
No resources found
kubectl create ing test -n ingress-nginx --rule test.example.com/"*"=test:80 --annotation=kubernetes.io/ingress.class=wrong
Name:             test
Labels:           <none>
Namespace:        ingress-nginx
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host              Path  Backends
  ----              ----  --------
  test.example.com
                    /   test:80 (<error: endpoints "test" not found>)
Annotations:        kubernetes.io/ingress.class: wrong
Events:             <none>
Name:             test
Labels:           <none>
Namespace:        ingress-nginx
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host              Path  Backends
  ----              ----  --------
  test.example.com
                    /   changed:80 (<error: endpoints "changed" not found>)
Annotations:        kubernetes.io/ingress.class: wrong
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    3s    nginx-ingress-controller  Scheduled for sync
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W1220 16:48:42.826798       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1220 16:48:42.826925       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I1220 16:48:42.830586       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="25" git="v1.25.3" state="clean" commit="434bfd82814af038ad94d62ebe59b133fcb50506" platform="linux/arm64"
I1220 16:48:42.959344       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
W1220 16:48:42.960797       7 main.go:111] No permissions to list and get Ingress Classes: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope, IngressClass feature will be disabled
W1220 16:48:42.964522       7 main.go:125] Unable to get NODE information: nodes "minikube" is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot get resource "nodes" in API group "" at the cluster scope
I1220 16:48:43.001544       7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1220 16:48:43.010706       7 nginx.go:260] "Starting NGINX Ingress controller"
I1220 16:48:43.014006       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a642aff7-7f7a-49aa-a490-9026abfd7a5c", APIVersion:"v1", ResourceVersion:"82328", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1220 16:48:44.212559       7 nginx.go:303] "Starting NGINX process"
I1220 16:48:44.212625       7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I1220 16:48:44.212895       7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1220 16:48:44.213131       7 controller.go:190] "Configuration changes detected, backend reload required"
I1220 16:48:44.216557       7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader
I1220 16:48:44.216643       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-699745fc87-sbh4s"
I1220 16:48:44.249971       7 controller.go:210] "Backend successfully reloaded"
I1220 16:48:44.250072       7 controller.go:221] "Initial sync, sleeping for 1 second"
I1220 16:48:44.250126       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-699745fc87-sbh4s", UID:"078b0462-2edd-46d3-a62b-00f5e8a96e40", APIVersion:"v1", ResourceVersion:"82353", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1220 16:54:14.906608       7 controller.go:331] ignoring ingress test in ingress-nginx based on annotation : ingress class annotation is not equal to the expected by Ingress Controller
I1220 16:54:14.906650       7 main.go:107] "successfully validated configuration, accepting" ingress="ingress-nginx/test"
I1220 16:54:14.911561       7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="ingress-nginx/test" error="ingress class annotation is not equal to the expected by Ingress Controller"
W1220 16:55:10.704428       7 controller.go:331] ignoring ingress test in ingress-nginx based on annotation : ingress class annotation is not equal to the expected by Ingress Controller
I1220 16:55:10.704455       7 main.go:107] "successfully validated configuration, accepting" ingress="ingress-nginx/test"
W1220 16:55:10.707085       7 controller.go:1108] Error obtaining Endpoints for Service "ingress-nginx/changed": no object matching key "ingress-nginx/changed" in local store
I1220 16:55:10.707205       7 controller.go:190] "Configuration changes detected, backend reload required"
I1220 16:55:10.707097       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"test", UID:"a54708bd-55ea-41c6-a763-f5c06459beaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"82740", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1220 16:55:10.753886       7 controller.go:210] "Backend successfully reloaded"
I1220 16:55:10.754123       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-699745fc87-sbh4s", UID:"078b0462-2edd-46d3-a62b-00f5e8a96e40", APIVersion:"v1", ResourceVersion:"82353", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

At 16:55:10.704428 you see it ignore the newly created ingress as the annotation does not match that for the controller. At 16:55:10.707097 you see it being processed after the update.

longwuyuan commented 8 months ago

@davidcurrie thanks for the update.

This message 

event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"test", UID:"a54708bd-55ea-41c6-a763-f5c06459beaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"82740", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1220 16:55:10.753886       7 controller.go:210] "Backend successfully reloaded"

does not mean that a request was sent to the controller and the controller matched the rules fro the ingress object named as "test". But if you expected to never ever see the name of the ingress called "test" ever again, in any log message of the controller, then that is very deep dive.

If you want to do that deep dive to describe a problem or bug in the code, then please edit the above post and create the app backend and send requests, first with the backend as test and next with the backend as changed. Correspondingly edit the logs also. If the request gets proper response code of 200, then it is proof that that the ingress was processed by the controller.

On a completely different observation, even though you have explained already, I still do not understand the practical need to disable the ingressClassResource. I suspect that the practical use of that spec --set controller.ingressClassResource.enabled=false is in the context of multiple ingressClasses existing and one of them being disabled.

Secondly I suspect that the spec controller.scope.enabled=true is retained by legacy and needs a developer to comment on, but its current state and expected behavioral impact on the controller is not clear to me. We do not have much priority for limiting the scope of the controller to a namespace as all users have practically switched to specifying the ingressClassName to control which ingress is processed by which controller instance, in a multi instance cluster.

Please wait for comments from others and kindly join the community meeting.

I could be wrong but my summary is ;

  1. You don't want to use ingressClass annotation or ingressClassName field in a ingress object.
  2. You want a controller instance to watch ingresses only in one namespace. If above assumptions are correct, then I have nothing more to comment as both of them are complicated grey areas that need to be discussed in a zoom meeting etc
davidcurrie commented 8 months ago

I can assure you that, if the backend service did exist then the controller would have configured nginx to route to it. I'm just trying to keep the recreate as simple as possible.

The desire to not use IngressClass is precisely because we want to keep everything scoped to the namespace. An IngressClass is not scoped to the namespace, nor are the permissions required to view them.

As you'll see from the linked PR in the corda organisation, our workaround for this was to give the controller permission to read IngressClass (i.e. the equivalent of setting controller.scope.enabled=true but leaving rbac.scope=false). It doesn't matter that we're still using the annotation-based approach and that there actually no IngressClass involved, this is sufficient to make things work, as just having the ability to read the IngressClass resources is enough to send the update down a different code path where it does check the annotation. That's all that the proposed fix above does - change it so that the annotation is always checked, regardless of whether IngessClass can be read.

longwuyuan commented 8 months ago

there is a flag to the controller like "--watch-ingress-without-class" https://github.com/search?q=repo%3Akubernetes%2Fingress-nginx%20without-class&type=code

longwuyuan commented 4 days ago
davidcurrie commented 4 days ago

I won't rehash the problem description. It's a shame the original PR was not merged, as I believe it was the correct fix. I understand that this is an edge case, though, and if there is no intention of implementing this simple change imminently, I'd close this issue.

longwuyuan commented 4 days ago

@davidcurrie thank you for the updated comment. The namespace-scoping is closer to the Ingres-API unlike a fringe implication, so closing this issue is not fair. The problem thought is resources available to work on it. So lets leave this open to track namespace-scoping.