search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.41k stars 8.24k forks source link

Ingress-nginx reports an error after enabling enable-owasp-modsecurity-crs true #9842

Closed laiqinghua closed 1 year ago

laiqinghua commented 1 year ago

what did I do： I added a configuration in configmap, and rebuilt the pod

apiVersion: v1 data:
allow-snippet-annotations: "true" custom-http-errors: 404,403
enable-modsecurity: "true" enable-owasp-modsecurity-crs: "true" modsecurity-snippet: |
SecRuleEngine On
SecRequestBodyAccess On SecAuditEngine RelevantOnly SecAuditLogFormat JSON
SecDebugLog /tmp/modsec_debug.log Include /etc/nginx/modsecurity/modsecurity.conf Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf kind: ConfigMap
...........

The current situation： ingress-nginx-controller could not be ready

The pod log keeps reporting errors，I stripped out the duplicates and pasted it below：

Error: exit status 1 2023/04/10 14:10:40 [emerg] 128#128: "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg501913557:131 nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg501913557:131 nginx: configuration file /tmp/nginx/nginx-cfg501913557 test failed

E0410 14:10:40.710542 7 queue.go:130] "requeuing" err=<

    -------------------------------------------------------------------------------
    Error: exit status 1
    2023/04/10 14:10:40 [emerg] 128#128: "modsecurity_rules_file" directive Rule id: 901001 is duplicated
     in /tmp/nginx/nginx-cfg501913557:131
    nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated
     in /tmp/nginx/nginx-cfg501913557:131
    nginx: configuration file /tmp/nginx/nginx-cfg501913557 test failed

    -------------------------------------------------------------------------------

key="kube-system/ingress-nginx-controller-cc8b2" I0410 14:10:40.710733 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"ingress-nginx-controller-86454467dd-fsl44", UID:"ccae5bf2-e031-4710-9c4c-e4a4e84c2cc2", APIVersion:"v1", ResourceVersion:"487896", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX:

NGINX Ingress controller Release: v1.7.0 Build: 72ff21ed9e26cb969052c753633049ba8a87ecf9 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6

verison: Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+k3s1", GitCommit:"57e8adb524611d79c4e17c27f15c5066e54b0421", GitTreeState:"clean", BuildDate:"2023-03-27T21:40:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+k3s1", GitCommit:"57e8adb524611d79c4e17c27f15c5066e54b0421", GitTreeState:"clean", BuildDate:"2023-03-27T21:40:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"}

os: PRETTY_NAME="Ubuntu 22.04 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu

kernel: Linux jp4 5.15.0-1029-oracle #35-Ubuntu SMP Tue Jan 24 15:21:05 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Current State of the controller: Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.7.0 helm.sh/chart=ingress-nginx-4.6.0 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: kube-system Controller: k8s.io/ingress-nginx Events:

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 1 year ago

/rempve-kind bug

You have not answered most of the questions asked in the new issue template so readers here will have a hard time trying to get to any actionable item, based on just the arbitrary vague information you have posted as issue description.

There are some docs here, so it may help if you can comment if the documented procedure to use modsecuity module works or fails for you.

You have the option to look at a new issue template and edit your issue description here and answer all the questions asked in the new issue template.

longwuyuan commented 1 year ago

/remove-kind bug

laiqinghua commented 1 year ago

What happened: what did I do： I added a configuration in configmap, and rebuilt the pod apiVersion: v1 data: allow-snippet-annotations: "true" enable-modsecurity: "true" enable-owasp-modsecurity-crs: "true" modsecurity-snippet: | SecRuleEngine On SecRequestBodyAccess On SecAuditEngine RelevantOnly SecAuditLogFormat JSON SecDebugLog /tmp/modsec_debug.log Include /etc/nginx/modsecurity/modsecurity.conf Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx creationTimestamp: "2023-04-11T01:48:15Z" labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.7.0 helm.sh/chart: ingress-nginx-4.6.0 name: ingress-nginx-controller namespace: ingress-nginx resourceVersion: "526377" uid: 1fd33c76-cdbf-434c-b34a-3d8ae0fabdf8

What you expected to happen:

There was a problem loading owasp rules in modsecurity

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller Release: v1.7.0 Build: 72ff21ed9e26cb969052c753633049ba8a87ecf9 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6 Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+k3s1", GitCommit:"57e8adb524611d79c4e17c27f15c5066e54b0421", GitTreeState:"clean", BuildDate:"2023-03-27T21:40:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+k3s1", GitCommit:"57e8adb524611d79c4e17c27f15c5066e54b0421", GitTreeState:"clean", BuildDate:"2023-03-27T21:40:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"} Environment:

ingress-nginx ingress-nginx 1 2023-04-11 09:48:09.94299908 +0800 CST deployed ingress-nginx-4.6.0 1.7.0

values is default

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller-admission ClusterIP 10.43.34.88 443/TCP 55m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.53.98 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 80:32358/TCP,443:30533/TCP 55m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 0/1 1 0 55m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-6c5dcd58d 1 1 0 55m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d root@jp4:/home/ubuntu/backup/ingress# kubectl -n ingress-nginx get all -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system pod/coredns-7b5bbc6644-df7m5 1/1 Running 0 7d17h 10.42.0.3 jp4 kube-system pod/local-path-provisioner-687d6d7765-b7t6m 1/1 Running 0 7d17h 10.42.0.2 jp4 kube-system pod/metrics-server-667586758d-hfsbm 1/1 Running 0 7d17h 10.42.0.4 jp4 cert-manager pod/cert-manager-6499989f7-6hx8x 1/1 Running 0 7d17h 10.42.1.7 jp3 cert-manager pod/cert-manager-cainjector-645b688547-f9rzl 1/1 Running 0 7d17h 10.42.1.6 jp3 cert-manager pod/cert-manager-webhook-6b7f49999f-m8bz4 1/1 Running 0 7d17h 10.42.1.8 jp3 default pod/serverstatus-b59494f8d-mhx5n 1/1 Running 0 6d22h 10.42.2.6 jp2 kube-system pod/svclb-serverstatusdata-071eac07-hlc6h 1/1 Running 0 6d22h 10.42.0.10 jp4 kube-system pod/svclb-serverstatusdata-071eac07-ccscn 1/1 Running 0 6d22h 10.42.1.12 jp3 kube-system pod/svclb-serverstatusdata-071eac07-nzzbw 1/1 Running 0 6d22h 10.42.2.7 jp2 kube-system pod/svclb-serverstatusdata-071eac07-qcfv5 1/1 Running 0 6d22h 10.42.3.5 jp1 default pod/oraclepolicy-767467b9dd-lf2gk 1/1 Running 0 6d19h 10.42.3.6 jp1 default pod/oraclepolicy-767467b9dd-dsshj 1/1 Running 0 6d19h 10.42.2.8 jp2 default pod/oraclepolicyarm-54c4896c74-8psx2 1/1 Running 0 6d19h 10.42.1.20 jp3 default pod/oraclepolicyarm-54c4896c74-q6rr6 1/1 Running 0 6d19h 10.42.0.11 jp4 syslog-ng pod/syslog-ng-6b5b545d4c-2hpnk 1/1 Running 0 6d17h 10.42.3.7 jp1 default pod/halo-695fd57bbd-zvdl7 1/1 Running 0 3d18h 10.42.1.30 jp3 default pod/custom-http-backend-f78d565b9-9s647 1/1 Running 0 17h 10.42.2.10 jp2 kube-system pod/svclb-ingress-nginx-controller-b29871b0-fnxf4 2/2 Running 0 56m 10.42.1.41 jp3 kube-system pod/svclb-ingress-nginx-controller-b29871b0-mhcrq 2/2 Running 0 56m 10.42.0.33 jp4 kube-system pod/svclb-ingress-nginx-controller-b29871b0-t8mbv 2/2 Running 0 56m 10.42.3.8 jp1 kube-system pod/svclb-ingress-nginx-controller-b29871b0-p5zws 2/2 Running 0 56m 10.42.2.11 jp2 ingress-nginx pod/ingress-nginx-controller-6c5dcd58d-vk849 0/1 CrashLoopBackOff 7 (51s ago) 14m 10.42.0.38 jp4

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.43.0.1 443/TCP 7d17h kube-system service/kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 7d17h k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.43.51.231 443/TCP 7d17h k8s-app=metrics-server cert-manager service/cert-manager ClusterIP 10.43.26.4 9402/TCP 7d17h app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager cert-manager service/cert-manager-webhook ClusterIP 10.43.27.53 443/TCP 7d17h app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook default service/halo ClusterIP 10.43.242.19 8090/TCP 6d22h app=halo default service/serverstatus ClusterIP 10.43.206.103 80/TCP 6d22h app=serverstatus default service/serverstatusdata LoadBalancer 10.43.54.165 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 35601:32767/TCP 6d22h app=serverstatus syslog-ng service/syslog-ng ClusterIP 10.43.127.172 514/TCP,514/UDP 6d17h app=syslog-ng default service/custom-http-backend ClusterIP 10.43.140.244 80/TCP 22h app=custom-http-backend ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.43.34.88 443/TCP 56m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.53.98 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 80:32358/TCP,443:30533/TCP 56m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/svclb-serverstatusdata-071eac07 4 4 4 4 4 6d22h lb-tcp-35601 rancher/klipper-lb:v0.4.0 app=svclb-serverstatusdata-071eac07 kube-system daemonset.apps/svclb-ingress-nginx-controller-b29871b0 4 4 4 4 4 56m lb-tcp-80,lb-tcp-443 rancher/klipper-lb:v0.4.0,rancher/klipper-lb:v0.4.0 app=svclb-ingress-nginx-controller-b29871b0

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR kube-system deployment.apps/coredns 1/1 1 1 7d17h coredns rancher/mirrored-coredns-coredns:1.9.4 k8s-app=kube-dns kube-system deployment.apps/local-path-provisioner 1/1 1 1 7d17h local-path-provisioner rancher/local-path-provisioner:v0.0.23 app=local-path-provisioner kube-system deployment.apps/metrics-server 1/1 1 1 7d17h metrics-server rancher/mirrored-metrics-server:v0.6.2 k8s-app=metrics-server cert-manager deployment.apps/cert-manager 1/1 1 1 7d17h cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager cert-manager deployment.apps/cert-manager-cainjector 1/1 1 1 7d17h cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.11.0 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector cert-manager deployment.apps/cert-manager-webhook 1/1 1 1 7d17h cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.11.0 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook default deployment.apps/serverstatus 1/1 1 1 6d22h container-0 cppla/serverstatus:latest workload.user.cattle.io/workloadselector=apps.deployment-default-serverstatus default deployment.apps/oraclepolicy 2/2 2 2 6d22h container-0 l931782512/oracle:latest workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default deployment.apps/oraclepolicyarm 2/2 2 2 6d22h container-0 l931782512/oracle:armlatest workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy syslog-ng deployment.apps/syslog-ng 1/1 1 1 6d17h syslog-ng balabit/syslog-ng:latest app=syslog-ng default deployment.apps/halo 1/1 1 1 6d22h container-0 halohub/halo:2.3 workload.user.cattle.io/workloadselector=apps.deployment-default-halo default deployment.apps/custom-http-backend 1/1 1 1 22h custom-http-backend vietanhs0817/nginx-errors:latest app=custom-http-backend ingress-nginx deployment.apps/ingress-nginx-controller 0/1 1 0 56m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR kube-system replicaset.apps/coredns-7b5bbc6644 1 1 1 7d17h coredns rancher/mirrored-coredns-coredns:1.9.4 k8s-app=kube-dns,pod-template-hash=7b5bbc6644 kube-system replicaset.apps/local-path-provisioner-687d6d7765 1 1 1 7d17h local-path-provisioner rancher/local-path-provisioner:v0.0.23 app=local-path-provisioner,pod-template-hash=687d6d7765 kube-system replicaset.apps/metrics-server-667586758d 1 1 1 7d17h metrics-server rancher/mirrored-metrics-server:v0.6.2 k8s-app=metrics-server,pod-template-hash=667586758d cert-manager replicaset.apps/cert-manager-6499989f7 1 1 1 7d17h cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=6499989f7 cert-manager replicaset.apps/cert-manager-cainjector-645b688547 1 1 1 7d17h cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.11.0 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector,pod-template-hash=645b688547 cert-manager replicaset.apps/cert-manager-webhook-6b7f49999f 1 1 1 7d17h cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.11.0 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook,pod-template-hash=6b7f49999f default replicaset.apps/serverstatus-b59494f8d 1 1 1 6d22h container-0 cppla/serverstatus:latest pod-template-hash=b59494f8d,workload.user.cattle.io/workloadselector=apps.deployment-default-serverstatus default replicaset.apps/oraclepolicy-767467b9dd 2 2 2 6d19h container-0 l931782512/oracle:latest pod-template-hash=767467b9dd,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default replicaset.apps/oraclepolicy-7b5d5f6544 0 0 0 6d22h container-0 l931782512/oracle:latest pod-template-hash=7b5d5f6544,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default replicaset.apps/oraclepolicyarm-54c4896c74 2 2 2 6d19h container-0 l931782512/oracle:armlatest pod-template-hash=54c4896c74,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default replicaset.apps/oraclepolicyarm-9cb64c65d 0 0 0 6d22h container-0 l931782512/oracle:armlatest pod-template-hash=9cb64c65d,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy syslog-ng replicaset.apps/syslog-ng-6b5b545d4c 1 1 1 6d17h syslog-ng balabit/syslog-ng:latest app=syslog-ng,pod-template-hash=6b5b545d4c default replicaset.apps/halo-695fd57bbd 1 1 1 3d18h container-0 halohub/halo:2.3 pod-template-hash=695fd57bbd,workload.user.cattle.io/workloadselector=apps.deployment-default-halo default replicaset.apps/halo-6bfb55d668 0 0 0 6d22h container-0 halohub/halo:2.1.0 pod-template-hash=6bfb55d668,workload.user.cattle.io/workloadselector=apps.deployment-default-halo default replicaset.apps/custom-http-backend-7d9967b765 0 0 0 22h custom-http-backend nginx app=custom-http-backend,pod-template-hash=7d9967b765 default replicaset.apps/custom-http-backend-67fb8b4799 0 0 0 20h custom-http-backend inanimate/echo-server app=custom-http-backend,pod-template-hash=67fb8b4799 default replicaset.apps/custom-http-backend-6d7f449f45 0 0 0 22h custom-http-backend nginx app=custom-http-backend,pod-template-hash=6d7f449f45 default replicaset.apps/custom-http-backend-6bbcdccbcf 0 0 0 20h custom-http-backend nginx app=custom-http-backend,pod-template-hash=6bbcdccbcf default replicaset.apps/custom-http-backend-65b985747c 0 0 0 20h custom-http-backend inanimate/echo-server app=custom-http-backend,pod-template-hash=65b985747c default replicaset.apps/custom-http-backend-f78d565b9 1 1 1 17h custom-http-backend vietanhs0817/nginx-errors:latest app=custom-http-backend,pod-template-hash=f78d565b9 default replicaset.apps/custom-http-backend-64c74cf76f 0 0 0 20h custom-http-backend inanimate/echo-server app=custom-http-backend,pod-template-hash=64c74cf76f ingress-nginx replicaset.apps/ingress-nginx-controller-6c5dcd58d 1 1 0 56m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d

Warning RELOAD 13m nginx-ingress-controller Error reloading NGINX:

Error: exit status 1 2023/04/11 02:33:35 [emerg] 127#127: "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg1904670917:131 nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg1904670917:131 nginx: configuration file /tmp/nginx/nginx-cfg1904670917 test failed

long log

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller-admission ClusterIP 10.43.34.88 443/TCP 62m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.53.98 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 80:32358/TCP,443:30533/TCP 62m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 0/1 1 0 62m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-6c5dcd58d 1 1 0 62m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d long log

How to reproduce this issue:

  1. install k3s
    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.24.12+k3s1 sh -s - --disable traefik
  2. install helm and configure
  3. helm install ingress-nginx helm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx \ --namespace ingress-nginx --create-namespace
  4. edit the ingress configuration in configmap kubectl edit -n ingress-nginx configmaps ingress-nginx-controller

apiVersion: v1 data: allow-snippet-annotations: "true" enable-modsecurity: "true" enable-owasp-modsecurity-crs: "true" modsecurity-snippet: | SecRuleEngine On SecRequestBodyAccess On SecAuditEngine RelevantOnly SecAuditLogFormat JSON SecDebugLog /tmp/modsec_debug.log Include /etc/nginx/modsecurity/modsecurity.conf Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx creationTimestamp: "2023-04-11T01:48:15Z" labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.7.0 helm.sh/chart: ingress-nginx-4.6.0 name: ingress-nginx-controller namespace: ingress-nginx

  1. delete ingress-nginx-controller pod， The rebuilding process will report an error。 kubectl logs -n ingress-nginx ingress-nginx-controller-xxxxx @longwuyuan
laiqinghua commented 1 year ago

@longwuyuan 兄弟辛苦帮忙看看 ，最后有复现文档，这个大概复现只需要3分钟。 感谢

longwuyuan commented 1 year ago
laiqinghua commented 1 year ago
errors detail root@jp4:/home/ubuntu/backup/ingress# kubectl logs -n ingress-nginx ingress-nginx-controller-6c5dcd58d-vk849 ------------------------------------------------------------------------------- NGINX Ingress controller Release: v1.7.0 Build: 72ff21ed9e26cb969052c753633049ba8a87ecf9 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6 ------------------------------------------------------------------------------- W0411 02:30:00.066767 6 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0411 02:30:00.066953 6 main.go:209] "Creating API client" host="https://10.43.0.1:443" I0411 02:30:00.073583 6 main.go:253] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.12+k3s1" state="clean" commit="57e8adb524611d79c4e17c27f15c5066e54b0421" platform="linux/arm64" I0411 02:30:00.246099 6 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem" I0411 02:30:00.261063 6 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key" I0411 02:30:00.268960 6 nginx.go:261] "Starting NGINX Ingress controller" I0411 02:30:00.297153 6 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"1fd33c76-cdbf-434c-b34a-3d8ae0fabdf8", APIVersion:"v1", ResourceVersion:"526377", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller I0411 02:30:01.374389 6 store.go:433] "Found valid IngressClass" ingress="default/halo" ingressclass="nginx" I0411 02:30:01.375156 6 backend_ssl.go:67] "Adding secret to local store" name="default/blog-91wan-top-tls" I0411 02:30:01.375178 6 store.go:433] "Found valid IngressClass" ingress="default/serverstatus" ingressclass="nginx" I0411 02:30:01.375837 6 backend_ssl.go:67] "Adding secret to local store" name="default/moniter-91wan-top-tls" I0411 02:30:01.376058 6 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"halo", UID:"5d6c3d6a-ae6b-4abd-a2eb-13dcf220aa71", APIVersion:"networking.k8s.io/v1", ResourceVersion:"525088", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync I0411 02:30:01.376070 6 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"serverstatus", UID:"93a434c5-8a27-43b9-99c1-2b73ed45a552", APIVersion:"networking.k8s.io/v1", ResourceVersion:"525089", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync I0411 02:30:01.470724 6 nginx.go:304] "Starting NGINX process" I0411 02:30:01.470925 6 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader... I0411 02:30:01.471139 6 nginx.go:324] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key" I0411 02:30:01.471644 6 controller.go:189] "Configuration changes detected, backend reload required" I0411 02:30:01.480980 6 status.go:84] "New leader elected" identity="ingress-nginx-controller-6c5dcd58d-nl2fx" E0411 02:30:02.069753 6 controller.go:201] Unexpected failure reloading the backend: ------------------------------------------------------------------------------- Error: exit status 1 2023/04/11 02:30:01 [emerg] 22#22: "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg2096902578:131 nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg2096902578:131 nginx: configuration file /tmp/nginx/nginx-cfg2096902578 test failed ------------------------------------------------------------------------------- E0411 02:30:02.069821 6 queue.go:130] "requeuing" err=< ------------------------------------------------------------------------------- Error: exit status 1 2023/04/11 02:30:01 [emerg] 22#22: "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg2096902578:131 nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg2096902578:131 nginx: configuration file /tmp/nginx/nginx-cfg2096902578 test failed ------------------------------------------------------------------------------- > key="initial-sync" I0411 02:30:02.070247 6 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-6c5dcd58d-vk849", UID:"de744bca-9cec-49d5-9eb6-f1c17f865591", APIVersion:"v1", ResourceVersion:"526398", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: ------------------------------------------------------------------------------- Error: exit status 1

What happened: what did I do： I added a configuration in configmap, and rebuilt the pod

configmap detail apiVersion: v1 data: allow-snippet-annotations: "true" enable-modsecurity: "true" enable-owasp-modsecurity-crs: "true" modsecurity-snippet: | SecRuleEngine On SecRequestBodyAccess On SecAuditEngine RelevantOnly SecAuditLogFormat JSON SecDebugLog /tmp/modsec_debug.log Include /etc/nginx/modsecurity/modsecurity.conf Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.7.0 helm.sh/chart: ingress-nginx-4.6.0 name: ingress-nginx-controller namespace: ingress-nginx

What you expected to happen:

There was a problem loading owasp rules in modsecurity

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller Release: v1.7.0 Build: 72ff21ed9e26cb969052c753633049ba8a87ecf9 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+k3s1", GitCommit:"57e8adb524611d79c4e17c27f15c5066e54b0421", GitTreeState:"clean", BuildDate:"2023-03-27T21:40:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.12+k3s1", GitCommit:"57e8adb524611d79c4e17c27f15c5066e54b0421", GitTreeState:"clean", BuildDate:"2023-03-27T21:40:47Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"}

Environment:

ingress-nginx ingress-nginx 1 2023-04-11 09:48:09.94299908 +0800 CST deployed ingress-nginx-4.6.0 1.7.0

values is default

get all -A -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/ingress-nginx-controller-6c5dcd58d-vk849 0/1 CrashLoopBackOff 7 (42s ago) 14m 10.42.0.38 jp4 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller-admission ClusterIP 10.43.34.88 443/TCP 55m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.53.98 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 80:32358/TCP,443:30533/TCP 55m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 0/1 1 0 55m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-6c5dcd58d 1 1 0 55m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d root@jp4:/home/ubuntu/backup/ingress# kubectl -n ingress-nginx get all -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system pod/coredns-7b5bbc6644-df7m5 1/1 Running 0 7d17h 10.42.0.3 jp4 kube-system pod/local-path-provisioner-687d6d7765-b7t6m 1/1 Running 0 7d17h 10.42.0.2 jp4 kube-system pod/metrics-server-667586758d-hfsbm 1/1 Running 0 7d17h 10.42.0.4 jp4 cert-manager pod/cert-manager-6499989f7-6hx8x 1/1 Running 0 7d17h 10.42.1.7 jp3 cert-manager pod/cert-manager-cainjector-645b688547-f9rzl 1/1 Running 0 7d17h 10.42.1.6 jp3 cert-manager pod/cert-manager-webhook-6b7f49999f-m8bz4 1/1 Running 0 7d17h 10.42.1.8 jp3 default pod/serverstatus-b59494f8d-mhx5n 1/1 Running 0 6d22h 10.42.2.6 jp2 kube-system pod/svclb-serverstatusdata-071eac07-hlc6h 1/1 Running 0 6d22h 10.42.0.10 jp4 kube-system pod/svclb-serverstatusdata-071eac07-ccscn 1/1 Running 0 6d22h 10.42.1.12 jp3 kube-system pod/svclb-serverstatusdata-071eac07-nzzbw 1/1 Running 0 6d22h 10.42.2.7 jp2 kube-system pod/svclb-serverstatusdata-071eac07-qcfv5 1/1 Running 0 6d22h 10.42.3.5 jp1 default pod/oraclepolicy-767467b9dd-lf2gk 1/1 Running 0 6d19h 10.42.3.6 jp1 default pod/oraclepolicy-767467b9dd-dsshj 1/1 Running 0 6d19h 10.42.2.8 jp2 default pod/oraclepolicyarm-54c4896c74-8psx2 1/1 Running 0 6d19h 10.42.1.20 jp3 default pod/oraclepolicyarm-54c4896c74-q6rr6 1/1 Running 0 6d19h 10.42.0.11 jp4 syslog-ng pod/syslog-ng-6b5b545d4c-2hpnk 1/1 Running 0 6d17h 10.42.3.7 jp1 default pod/halo-695fd57bbd-zvdl7 1/1 Running 0 3d18h 10.42.1.30 jp3 default pod/custom-http-backend-f78d565b9-9s647 1/1 Running 0 17h 10.42.2.10 jp2 kube-system pod/svclb-ingress-nginx-controller-b29871b0-fnxf4 2/2 Running 0 56m 10.42.1.41 jp3 kube-system pod/svclb-ingress-nginx-controller-b29871b0-mhcrq 2/2 Running 0 56m 10.42.0.33 jp4 kube-system pod/svclb-ingress-nginx-controller-b29871b0-t8mbv 2/2 Running 0 56m 10.42.3.8 jp1 kube-system pod/svclb-ingress-nginx-controller-b29871b0-p5zws 2/2 Running 0 56m 10.42.2.11 jp2 ingress-nginx pod/ingress-nginx-controller-6c5dcd58d-vk849 0/1 CrashLoopBackOff 7 (51s ago) 14m 10.42.0.38 jp4 NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.43.0.1 443/TCP 7d17h kube-system service/kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 7d17h k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.43.51.231 443/TCP 7d17h k8s-app=metrics-server cert-manager service/cert-manager ClusterIP 10.43.26.4 9402/TCP 7d17h app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager cert-manager service/cert-manager-webhook ClusterIP 10.43.27.53 443/TCP 7d17h app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook default service/halo ClusterIP 10.43.242.19 8090/TCP 6d22h app=halo default service/serverstatus ClusterIP 10.43.206.103 80/TCP 6d22h app=serverstatus default service/serverstatusdata LoadBalancer 10.43.54.165 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 35601:32767/TCP 6d22h app=serverstatus syslog-ng service/syslog-ng ClusterIP 10.43.127.172 514/TCP,514/UDP 6d17h app=syslog-ng default service/custom-http-backend ClusterIP 10.43.140.244 80/TCP 22h app=custom-http-backend ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.43.34.88 443/TCP 56m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.53.98 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 80:32358/TCP,443:30533/TCP 56m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/svclb-serverstatusdata-071eac07 4 4 4 4 4 6d22h lb-tcp-35601 rancher/klipper-lb:v0.4.0 app=svclb-serverstatusdata-071eac07 kube-system daemonset.apps/svclb-ingress-nginx-controller-b29871b0 4 4 4 4 4 56m lb-tcp-80,lb-tcp-443 rancher/klipper-lb:v0.4.0,rancher/klipper-lb:v0.4.0 app=svclb-ingress-nginx-controller-b29871b0 NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR kube-system deployment.apps/coredns 1/1 1 1 7d17h coredns rancher/mirrored-coredns-coredns:1.9.4 k8s-app=kube-dns kube-system deployment.apps/local-path-provisioner 1/1 1 1 7d17h local-path-provisioner rancher/local-path-provisioner:v0.0.23 app=local-path-provisioner kube-system deployment.apps/metrics-server 1/1 1 1 7d17h metrics-server rancher/mirrored-metrics-server:v0.6.2 k8s-app=metrics-server cert-manager deployment.apps/cert-manager 1/1 1 1 7d17h cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager cert-manager deployment.apps/cert-manager-cainjector 1/1 1 1 7d17h cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.11.0 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector cert-manager deployment.apps/cert-manager-webhook 1/1 1 1 7d17h cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.11.0 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook default deployment.apps/serverstatus 1/1 1 1 6d22h container-0 cppla/serverstatus:latest workload.user.cattle.io/workloadselector=apps.deployment-default-serverstatus default deployment.apps/oraclepolicy 2/2 2 2 6d22h container-0 l931782512/oracle:latest workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default deployment.apps/oraclepolicyarm 2/2 2 2 6d22h container-0 l931782512/oracle:armlatest workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy syslog-ng deployment.apps/syslog-ng 1/1 1 1 6d17h syslog-ng balabit/syslog-ng:latest app=syslog-ng default deployment.apps/halo 1/1 1 1 6d22h container-0 halohub/halo:2.3 workload.user.cattle.io/workloadselector=apps.deployment-default-halo default deployment.apps/custom-http-backend 1/1 1 1 22h custom-http-backend vietanhs0817/nginx-errors:latest app=custom-http-backend ingress-nginx deployment.apps/ingress-nginx-controller 0/1 1 0 56m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR kube-system replicaset.apps/coredns-7b5bbc6644 1 1 1 7d17h coredns rancher/mirrored-coredns-coredns:1.9.4 k8s-app=kube-dns,pod-template-hash=7b5bbc6644 kube-system replicaset.apps/local-path-provisioner-687d6d7765 1 1 1 7d17h local-path-provisioner rancher/local-path-provisioner:v0.0.23 app=local-path-provisioner,pod-template-hash=687d6d7765 kube-system replicaset.apps/metrics-server-667586758d 1 1 1 7d17h metrics-server rancher/mirrored-metrics-server:v0.6.2 k8s-app=metrics-server,pod-template-hash=667586758d cert-manager replicaset.apps/cert-manager-6499989f7 1 1 1 7d17h cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=6499989f7 cert-manager replicaset.apps/cert-manager-cainjector-645b688547 1 1 1 7d17h cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.11.0 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector,pod-template-hash=645b688547 cert-manager replicaset.apps/cert-manager-webhook-6b7f49999f 1 1 1 7d17h cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.11.0 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook,pod-template-hash=6b7f49999f default replicaset.apps/serverstatus-b59494f8d 1 1 1 6d22h container-0 cppla/serverstatus:latest pod-template-hash=b59494f8d,workload.user.cattle.io/workloadselector=apps.deployment-default-serverstatus default replicaset.apps/oraclepolicy-767467b9dd 2 2 2 6d19h container-0 l931782512/oracle:latest pod-template-hash=767467b9dd,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default replicaset.apps/oraclepolicy-7b5d5f6544 0 0 0 6d22h container-0 l931782512/oracle:latest pod-template-hash=7b5d5f6544,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default replicaset.apps/oraclepolicyarm-54c4896c74 2 2 2 6d19h container-0 l931782512/oracle:armlatest pod-template-hash=54c4896c74,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy default replicaset.apps/oraclepolicyarm-9cb64c65d 0 0 0 6d22h container-0 l931782512/oracle:armlatest pod-template-hash=9cb64c65d,workload.user.cattle.io/workloadselector=apps.deployment-default-oraclepolicy syslog-ng replicaset.apps/syslog-ng-6b5b545d4c 1 1 1 6d17h syslog-ng balabit/syslog-ng:latest app=syslog-ng,pod-template-hash=6b5b545d4c default replicaset.apps/halo-695fd57bbd 1 1 1 3d18h container-0 halohub/halo:2.3 pod-template-hash=695fd57bbd,workload.user.cattle.io/workloadselector=apps.deployment-default-halo default replicaset.apps/halo-6bfb55d668 0 0 0 6d22h container-0 halohub/halo:2.1.0 pod-template-hash=6bfb55d668,workload.user.cattle.io/workloadselector=apps.deployment-default-halo default replicaset.apps/custom-http-backend-7d9967b765 0 0 0 22h custom-http-backend nginx app=custom-http-backend,pod-template-hash=7d9967b765 default replicaset.apps/custom-http-backend-67fb8b4799 0 0 0 20h custom-http-backend inanimate/echo-server app=custom-http-backend,pod-template-hash=67fb8b4799 default replicaset.apps/custom-http-backend-6d7f449f45 0 0 0 22h custom-http-backend nginx app=custom-http-backend,pod-template-hash=6d7f449f45 default replicaset.apps/custom-http-backend-6bbcdccbcf 0 0 0 20h custom-http-backend nginx app=custom-http-backend,pod-template-hash=6bbcdccbcf default replicaset.apps/custom-http-backend-65b985747c 0 0 0 20h custom-http-backend inanimate/echo-server app=custom-http-backend,pod-template-hash=65b985747c default replicaset.apps/custom-http-backend-f78d565b9 1 1 1 17h custom-http-backend vietanhs0817/nginx-errors:latest app=custom-http-backend,pod-template-hash=f78d565b9 default replicaset.apps/custom-http-backend-64c74cf76f 0 0 0 20h custom-http-backend inanimate/echo-server app=custom-http-backend,pod-template-hash=64c74cf76f ingress-nginx replicaset.apps/ingress-nginx-controller-6c5dcd58d 1 1 0 56m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d
kubectl -n ingress-nginx describe pod ingress-nginx-controller-6c5dcd58d-vk849 ------------------------------------------------------------------------------- Warning RELOAD 13m nginx-ingress-controller Error reloading NGINX: ------------------------------------------------------------------------------- Error: exit status 1 2023/04/11 02:33:32 [emerg] 22#22: "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg4080343104:131 nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg4080343104:131 nginx: configuration file /tmp/nginx/nginx-cfg4080343104 test failed ------------------------------------------------------------------------------- Warning RELOAD 13m nginx-ingress-controller Error reloading NGINX: ------------------------------------------------------------------------------- Error: exit status 1 2023/04/11 02:33:35 [emerg] 127#127: "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg1904670917:131 nginx: [emerg] "modsecurity_rules_file" directive Rule id: 901001 is duplicated in /tmp/nginx/nginx-cfg1904670917:131 nginx: configuration file /tmp/nginx/nginx-cfg1904670917 test failed
kubectl -n ingress-nginx describe svc ingress-nginx-controller Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.7.0 helm.sh/chart=ingress-nginx-4.6.0 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.43.53.98 IPs: 10.43.53.98 LoadBalancer Ingress: 10.0.0.190, 10.0.0.191, 10.0.0.195, 10.0.0.29 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32358/TCP Endpoints: Port: https 443/TCP TargetPort: https/TCP NodePort: https 30533/TCP Endpoints: Session Affinity: None External Traffic Policy: Cluster Events:
- **Current state of ingress object, if applicable**: - `kubectl -n get all,ing -o wide`
kubectl -n ingress-nginx get all,ing -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/ingress-nginx-controller-6c5dcd58d-vk849 0/1 CrashLoopBackOff 9 (19s ago) 21m 10.42.0.38 jp4 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller-admission ClusterIP 10.43.34.88 443/TCP 62m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.53.98 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 80:32358/TCP,443:30533/TCP 62m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 0/1 1 0 62m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-6c5dcd58d 1 1 0 62m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d
- `kubectl -n describe ing `
kubectl -n ingress-nginx describe ingress halo Name: halo Labels: Namespace: default Address: 10.0.0.190,10.0.0.191,10.0.0.195,10.0.0.29 Ingress Class: Default backend: TLS: blog-91wan-top-tls terminates blog.91wan.top Rules: Host Path Backends ---- ---- -------- blog.91wan.top / halo:haloweb (10.42.1.30:8090) Annotations: cert-manager.io/issuer: letsencrypt-prod kubernetes.io/ingress.class: nginx
**How to reproduce this issue**:
reproduce this issue 1. install k3s curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.24.12+k3s1 sh -s - --disable traefik 2. install helm and configure 3. helm install ingress-nginx helm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx \ --namespace ingress-nginx --create-namespace 4. edit the ingress configuration in configmap kubectl edit -n ingress-nginx configmaps ingress-nginx-controller apiVersion: v1 data: allow-snippet-annotations: "true" **enable-modsecurity: "true" enable-owasp-modsecurity-crs: "true" modsecurity-snippet: | SecRuleEngine On SecRequestBodyAccess On SecAuditEngine RelevantOnly SecAuditLogFormat JSON SecDebugLog /tmp/modsec_debug.log Include /etc/nginx/modsecurity/modsecurity.conf Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf** kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx creationTimestamp: "2023-04-11T01:48:15Z" labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.7.0 helm.sh/chart: ingress-nginx-4.6.0 name: ingress-nginx-controller namespace: ingress-nginx 5. delete ingress-nginx-controller pod， The rebuilding process will report an error。 kubectl logs -n ingress-nginx ingress-nginx-controller-xxxxx
@longwuyuan
laiqinghua commented 1 year ago

@longwuyuan

longwuyuan commented 1 year ago

duplicate of https://github.com/kubernetes/ingress-nginx/issues/9847