search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.28k stars 8.21k forks source link

Multiple nginx-ingress cannot reconcile the related Ingress objects but all the ingress object #11746

Closed eminaktas closed 1 month ago

eminaktas commented 1 month ago

What happened:

We have two ingress-nginx controllers deployed in a namespace. One for internal, the other one is for external. When we deploy the ingress resources with related ingress class name, we see that both ingress controllers reconciles the both objects.

logs when I apply an ingress object (at the same time both deployments reports the same logs) ``` ingress-nginx-4-11-1-internal-controller-594984bbdd-f5xjk I0807 12:41:14.932245 7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/test" error="ingress d oes not contain a valid IngressClass" ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 I0807 12:41:14.932162 7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/test" error="ingress d oes not contain a valid IngressClass" ingress-nginx-4-11-1-internal-controller-594984bbdd-6vkbl I0807 12:41:14.932469 7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/test" error="ingress d oes not contain a valid IngressClass" ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 W0807 12:41:14.937048 7 controller.go:333] ignoring ingress internal-test in cloud-qa-f1 based on annotation : ingress does not contain a valid Ingr essClass ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 I0807 12:41:14.937062 7 main.go:107] "successfully validated configuration, accepting" ingress="cloud-qa-f1/internal-test" ingress-nginx-4-11-1-internal-controller-594984bbdd-6vkbl I0807 12:41:14.938739 7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/internal-test" error=" ingress does not contain a valid IngressClass" ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 I0807 12:41:14.938816 7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/internal-test" error=" ingress does not contain a valid IngressClass" ingress-nginx-4-11-1-internal-controller-594984bbdd-f5xjk I0807 12:41:14.938960 7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/internal-test" error=" ingress does not contain a valid IngressClass" ```

What you expected to happen:

Since we are deploying two different ingress classes with different controller value, controllers should only be applying the configuration from related Ingresses. This happens when each controller runs the sync process and reconciles both objects.

NAME            CLASS                        HOSTS                     ADDRESS         PORTS   AGE
internal-test   internal-nginx-cloud-qa-f1   internal-qa-f1.test.com   192.168.49.51   80      10m
test            nginx-cloud-qa-f1            qa-f1.test.com            192.168.49.51   80      10m

For example, for above resources, they get same IP. Even though, both controller has it own IP can their own ingress class definitions.

NAME                                                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                     AGE
ingress-nginx-4-11-1-controller                      LoadBalancer   10.99.186.39     192.168.49.50   80:32059/TCP,443:32554/TCP,6651:30647/TCP   100m
ingress-nginx-4-11-1-internal-controller             LoadBalancer   10.106.114.245   192.168.49.51   80:32378/TCP,443:30863/TCP                  99m

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.11.1
  Build:         7c44f992012555ff7f4e47c08d7c542ca9b4b1f7
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

Environment:

helm ls -A | grep -i ingress
ingress-nginx-4-11-1            cloud-qa-f1             1               2024-08-07 14:32:55.525478 +0300 +03    deployed        ingress-nginx-4.11.1    1.11.1
ingress-nginx-4-11-1-internal   cloud-qa-f1             1               2024-08-07 14:33:49.480643 +0300 +03    deployed        ingress-nginx-4.11.1    1.11.1
k8s-ci-robot commented 1 month ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
eminaktas commented 1 month ago
ingress-nginx-4-11-1 helm values ``` USER-SUPPLIED VALUES: commonLabels: {} controller: addHeaders: {} admissionWebhooks: annotations: {} certManager: admissionCert: duration: "" enabled: false rootCert: duration: "" certificate: /usr/local/certificates/cert createSecretJob: name: create resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault enabled: true existingPsp: "" extraEnvs: [] failurePolicy: Fail key: /usr/local/certificates/key labels: {} name: admission namespaceSelector: {} objectSelector: {} patch: enabled: true image: digest: sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 image: ingress-nginx/kube-webhook-certgen pullPolicy: IfNotPresent registry: registry.k8s.io tag: v1.4.1 labels: {} networkPolicy: enabled: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} priorityClassName: "" rbac: create: true securityContext: {} serviceAccount: automountServiceAccountToken: true create: true name: "" tolerations: [] patchWebhookJob: name: patch resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault port: 8443 service: annotations: {} externalIPs: [] loadBalancerSourceRanges: [] servicePort: 443 type: ClusterIP affinity: {} allowSnippetAnnotations: true annotations: {} autoscaling: annotations: {} behavior: scaleDown: policies: - periodSeconds: 120 type: Pods value: 1 selectPolicy: Disabled stabilizationWindowSeconds: 300 scaleUp: policies: - periodSeconds: 1 type: Pods value: 1 stabilizationWindowSeconds: 1 enabled: true maxReplicas: 12 minReplicas: 3 targetCPUUtilizationPercentage: 90 targetMemoryUtilizationPercentage: 90 autoscalingTemplate: [] config: max-worker-connections: "32768" proxy-body-size: 20m use-forwarded-headers: true worker-cpu-affinity: auto worker-processes: 4 configAnnotations: {} configMapNamespace: "" containerName: controller containerPort: http: 80 https: 443 containerSecurityContext: {} customTemplate: configMapKey: "" configMapName: "" disableLeaderElection: false dnsConfig: {} dnsPolicy: ClusterFirst electionID: ingress-controller-leader electionTTL: "" enableAnnotationValidations: false enableMimalloc: true enableTopologyAwareRouting: false existingPsp: "" extraArgs: enable-ssl-passthrough: true extraContainers: [] extraEnvs: [] extraInitContainers: [] extraModules: [] extraVolumeMounts: [] extraVolumes: [] healthCheckHost: "" healthCheckPath: /healthz hostAliases: [] hostNetwork: false hostPort: enabled: false ports: http: 80 https: 443 hostname: {} image: allowPrivilegeEscalation: false chroot: false digest: sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a digestChroot: sha256:7cabe4bd7558bfdf5b707976d7be56fd15ffece735d7c90fc238b6eda290fd8d image: ingress-nginx/controller pullPolicy: IfNotPresent readOnlyRootFilesystem: false registry: registry.k8s.io runAsNonRoot: true runAsUser: 101 seccompProfile: type: RuntimeDefault tag: v1.11.1 ingressClass: nginx-cloud-qa-f1 ingressClassByName: true ingressClassResource: aliases: [] annotations: {} controllerValue: k8s.io/external-ingress-nginx default: false enabled: true name: nginx-cloud-qa-f1 parameters: {} keda: apiVersion: keda.sh/v1alpha1 behavior: {} cooldownPeriod: 300 enabled: false maxReplicas: 11 minReplicas: 1 pollingInterval: 30 restoreToOriginalReplicaCount: false scaledObject: annotations: {} triggers: [] kind: Deployment labels: {} lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 maxmindLicenseKey: "" metrics: enabled: true port: 10254 portName: metrics prometheusRule: additionalLabels: {} enabled: false rules: [] service: annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" externalIPs: [] labels: {} loadBalancerSourceRanges: [] servicePort: 10254 type: ClusterIP serviceMonitor: additionalLabels: {} annotations: {} enabled: false metricRelabelings: [] namespace: "" namespaceSelector: {} relabelings: [] scrapeInterval: 30s targetLabels: [] minAvailable: 1 minReadySeconds: 0 name: controller networkPolicy: enabled: false nodeSelector: kubernetes.io/os: linux opentelemetry: containerSecurityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault enabled: false image: digest: sha256:13bee3f5223883d3ca62fee7309ad02d22ec00ff0d7033e3e9aca7a9f60fd472 distroless: true image: ingress-nginx/opentelemetry registry: registry.k8s.io tag: v20230721-3e2062ee5 name: opentelemetry resources: {} podAnnotations: {} podLabels: {} podSecurityContext: {} priorityClassName: "" proxySetHeaders: {} publishService: enabled: true pathOverride: "" readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 replicaCount: 1 reportNodeInternalIp: false resources: limits: cpu: 2 memory: 2.2Gi requests: cpu: 2 memory: 2.2Gi scope: enabled: true namespace: "" namespaceSelector: "" service: annotations: {} appProtocol: true clusterIP: "" enableHttp: true enableHttps: true enabled: true external: enabled: true externalIPs: [] externalTrafficPolicy: Local internal: annotations: {} appProtocol: true clusterIP: "" enabled: false externalIPs: [] externalTrafficPolicy: "" ipFamilies: - IPv4 ipFamilyPolicy: SingleStack loadBalancerClass: "" loadBalancerIP: "" loadBalancerSourceRanges: [] nodePorts: http: "" https: "" tcp: {} udp: {} ports: {} sessionAffinity: "" targetPorts: {} type: "" ipFamilies: - IPv4 ipFamilyPolicy: SingleStack labels: {} loadBalancerClass: "" loadBalancerIP: "" loadBalancerSourceRanges: [] nodePorts: http: "" https: "" tcp: {} udp: {} ports: http: 80 https: 443 sessionAffinity: "" targetPorts: http: http https: https type: LoadBalancer shareProcessNamespace: false sysctls: {} tcp: annotations: {} configMapNamespace: "" terminationGracePeriodSeconds: 300 tolerations: [] topologySpreadConstraints: [] udp: annotations: {} configMapNamespace: "" updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate watchIngressWithoutClass: false defaultBackend: affinity: {} autoscaling: annotations: {} enabled: true maxReplicas: 10 minReplicas: 1 targetCPUUtilizationPercentage: 50 targetMemoryUtilizationPercentage: 50 containerSecurityContext: {} enabled: false existingPsp: "" extraArgs: {} extraConfigMaps: [] extraEnvs: [] extraVolumeMounts: [] extraVolumes: [] image: allowPrivilegeEscalation: false image: defaultbackend-amd64 pullPolicy: IfNotPresent readOnlyRootFilesystem: true registry: registry.k8s.io runAsNonRoot: true runAsUser: 65534 seccompProfile: type: RuntimeDefault tag: "1.5" labels: {} livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 minAvailable: 1 minReadySeconds: 0 name: defaultbackend networkPolicy: enabled: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} podLabels: {} podSecurityContext: {} port: 8080 priorityClassName: "" readinessProbe: failureThreshold: 6 initialDelaySeconds: 0 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 5 replicaCount: 1 resources: {} service: annotations: {} externalIPs: [] loadBalancerSourceRanges: [] servicePort: 80 type: ClusterIP serviceAccount: automountServiceAccountToken: true create: true name: "" tolerations: [] topologySpreadConstraints: [] updateStrategy: {} dhParam: "" imagePullSecrets: [] namespaceOverride: "" podSecurityPolicy: enabled: false portNamePrefix: "" rbac: create: true scope: true revisionHistoryLimit: 10 serviceAccount: annotations: {} automountServiceAccountToken: true create: true name: "" tcp: {} udp: {} ```
ingress-nginx-4-11-1-internal helm values ``` USER-SUPPLIED VALUES: commonLabels: {} controller: addHeaders: {} admissionWebhooks: annotations: {} certManager: admissionCert: duration: "" enabled: false rootCert: duration: "" certificate: /usr/local/certificates/cert createSecretJob: name: create resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault enabled: true existingPsp: "" extraEnvs: [] failurePolicy: Fail key: /usr/local/certificates/key labels: {} name: admission namespaceSelector: {} objectSelector: {} patch: enabled: true image: digest: sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 image: ingress-nginx/kube-webhook-certgen pullPolicy: IfNotPresent registry: registry.k8s.io tag: v1.4.1 labels: {} networkPolicy: enabled: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} priorityClassName: "" rbac: create: true securityContext: {} serviceAccount: automountServiceAccountToken: true create: true name: "" tolerations: [] patchWebhookJob: name: patch resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault port: 8443 service: annotations: {} externalIPs: [] loadBalancerSourceRanges: [] servicePort: 443 type: ClusterIP affinity: {} allowSnippetAnnotations: true annotations: {} autoscaling: annotations: {} behavior: scaleDown: policies: - periodSeconds: 120 type: Pods value: 1 selectPolicy: Disabled stabilizationWindowSeconds: 300 scaleUp: policies: - periodSeconds: 1 type: Pods value: 1 stabilizationWindowSeconds: 1 enabled: true maxReplicas: 12 minReplicas: 3 targetCPUUtilizationPercentage: 90 targetMemoryUtilizationPercentage: 90 autoscalingTemplate: [] config: max-worker-connections: "32768" proxy-body-size: 20m use-forwarded-headers: true worker-cpu-affinity: auto worker-processes: 4 configAnnotations: {} configMapNamespace: "" containerName: controller containerPort: http: 80 https: 443 containerSecurityContext: {} customTemplate: configMapKey: "" configMapName: "" disableLeaderElection: false dnsConfig: {} dnsPolicy: ClusterFirst electionID: internal-ingress-controller-leader electionTTL: "" enableAnnotationValidations: false enableMimalloc: true enableTopologyAwareRouting: false existingPsp: "" extraArgs: enable-ssl-passthrough: true extraContainers: [] extraEnvs: [] extraInitContainers: [] extraModules: [] extraVolumeMounts: [] extraVolumes: [] healthCheckHost: "" healthCheckPath: /healthz hostAliases: [] hostNetwork: false hostPort: enabled: false ports: http: 80 https: 443 hostname: {} image: allowPrivilegeEscalation: false chroot: false digest: sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a digestChroot: sha256:7cabe4bd7558bfdf5b707976d7be56fd15ffece735d7c90fc238b6eda290fd8d image: ingress-nginx/controller pullPolicy: IfNotPresent readOnlyRootFilesystem: false registry: registry.k8s.io runAsNonRoot: true runAsUser: 101 seccompProfile: type: RuntimeDefault tag: v1.11.1 ingressClass: internal-nginx-cloud-qa-f1 ingressClassByName: true ingressClassResource: aliases: [] annotations: {} controllerValue: k8s.io/internal-ingress-nginx default: false enabled: true name: internal-nginx-cloud-qa-f1 parameters: {} keda: apiVersion: keda.sh/v1alpha1 behavior: {} cooldownPeriod: 300 enabled: false maxReplicas: 11 minReplicas: 1 pollingInterval: 30 restoreToOriginalReplicaCount: false scaledObject: annotations: {} triggers: [] kind: Deployment labels: {} lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 maxmindLicenseKey: "" metrics: enabled: true port: 10254 portName: metrics prometheusRule: additionalLabels: {} enabled: false rules: [] service: annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" externalIPs: [] labels: {} loadBalancerSourceRanges: [] servicePort: 10254 type: ClusterIP serviceMonitor: additionalLabels: {} annotations: {} enabled: false metricRelabelings: [] namespace: "" namespaceSelector: {} relabelings: [] scrapeInterval: 30s targetLabels: [] minAvailable: 1 minReadySeconds: 0 name: controller networkPolicy: enabled: false nodeSelector: kubernetes.io/os: linux opentelemetry: containerSecurityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault enabled: false image: digest: sha256:13bee3f5223883d3ca62fee7309ad02d22ec00ff0d7033e3e9aca7a9f60fd472 distroless: true image: ingress-nginx/opentelemetry registry: registry.k8s.io tag: v20230721-3e2062ee5 name: opentelemetry resources: {} podAnnotations: {} podLabels: {} podSecurityContext: {} priorityClassName: "" proxySetHeaders: {} publishService: enabled: true pathOverride: "" readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 replicaCount: 1 reportNodeInternalIp: false resources: limits: cpu: 2 memory: 2.2Gi requests: cpu: 2 memory: 2.2Gi scope: enabled: true namespace: "" namespaceSelector: "" service: annotations: {} appProtocol: true clusterIP: "" enableHttp: true enableHttps: true enabled: true external: enabled: true externalIPs: [] externalTrafficPolicy: Local internal: annotations: {} appProtocol: true clusterIP: "" enabled: false externalIPs: [] externalTrafficPolicy: "" ipFamilies: - IPv4 ipFamilyPolicy: SingleStack loadBalancerClass: "" loadBalancerIP: "" loadBalancerSourceRanges: [] nodePorts: http: "" https: "" tcp: {} udp: {} ports: {} sessionAffinity: "" targetPorts: {} type: "" ipFamilies: - IPv4 ipFamilyPolicy: SingleStack labels: {} loadBalancerClass: "" loadBalancerIP: "" loadBalancerSourceRanges: [] nodePorts: http: "" https: "" tcp: {} udp: {} ports: http: 80 https: 443 sessionAffinity: "" targetPorts: http: http https: https type: LoadBalancer shareProcessNamespace: false sysctls: {} tcp: annotations: {} configMapNamespace: "" terminationGracePeriodSeconds: 300 tolerations: [] topologySpreadConstraints: [] udp: annotations: {} configMapNamespace: "" updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate watchIngressWithoutClass: false defaultBackend: affinity: {} autoscaling: annotations: {} enabled: true maxReplicas: 10 minReplicas: 1 targetCPUUtilizationPercentage: 50 targetMemoryUtilizationPercentage: 50 containerSecurityContext: {} enabled: false existingPsp: "" extraArgs: {} extraConfigMaps: [] extraEnvs: [] extraVolumeMounts: [] extraVolumes: [] image: allowPrivilegeEscalation: false image: defaultbackend-amd64 pullPolicy: IfNotPresent readOnlyRootFilesystem: true registry: registry.k8s.io runAsNonRoot: true runAsUser: 65534 seccompProfile: type: RuntimeDefault tag: "1.5" labels: {} livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 minAvailable: 1 minReadySeconds: 0 name: defaultbackend networkPolicy: enabled: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} podLabels: {} podSecurityContext: {} port: 8080 priorityClassName: "" readinessProbe: failureThreshold: 6 initialDelaySeconds: 0 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 5 replicaCount: 1 resources: {} service: annotations: {} externalIPs: [] loadBalancerSourceRanges: [] servicePort: 80 type: ClusterIP serviceAccount: automountServiceAccountToken: true create: true name: "" tolerations: [] topologySpreadConstraints: [] updateStrategy: {} dhParam: "" imagePullSecrets: [] namespaceOverride: "" podSecurityPolicy: enabled: false portNamePrefix: "" rbac: create: true scope: true revisionHistoryLimit: 10 serviceAccount: annotations: {} automountServiceAccountToken: true create: true name: "" tcp: {} udp: {} ```
ingressclasses resources ``` Name: internal-nginx-cloud-qa-f1 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1-internal app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 Annotations: meta.helm.sh/release-name: ingress-nginx-4-11-1-internal meta.helm.sh/release-namespace: cloud-qa-f1 Controller: k8s.io/internal-ingress-nginx Events: Name: nginx-cloud-qa-f1 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 Annotations: meta.helm.sh/release-name: ingress-nginx-4-11-1 meta.helm.sh/release-namespace: cloud-qa-f1 Controller: k8s.io/external-ingress-nginx Events: ```
get all ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/ingress-nginx-4-11-1-controller-545b5f6457-428f5 1/1 Running 0 89m 10.244.2.16 minikube-m03 pod/ingress-nginx-4-11-1-controller-545b5f6457-bv9hq 1/1 Running 0 88m 10.244.0.12 minikube pod/ingress-nginx-4-11-1-controller-545b5f6457-bxkjz 1/1 Running 0 89m 10.244.1.14 minikube-m02 pod/ingress-nginx-4-11-1-internal-controller-594984bbdd-6vkbl 1/1 Running 0 90m 10.244.2.14 minikube-m03 pod/ingress-nginx-4-11-1-internal-controller-594984bbdd-f5xjk 1/1 Running 0 90m 10.244.0.10 minikube pod/ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 1/1 Running 0 90m 10.244.1.12 minikube-m02 pod/nginx 1/1 Running 0 85m 10.244.1.15 minikube-m02 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-4-11-1-controller LoadBalancer 10.99.186.39 192.168.49.50 80:32059/TCP,443:32554/TCP,6651:30647/TCP 91m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-4-11-1-controller-admission ClusterIP 10.111.93.139 443/TCP 91m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-4-11-1-controller-metrics ClusterIP 10.98.76.131 10254/TCP 91m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-4-11-1-internal-controller LoadBalancer 10.106.114.245 192.168.49.51 80:32378/TCP,443:30863/TCP 90m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1-internal,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-4-11-1-internal-controller-admission ClusterIP 10.107.120.250 443/TCP 90m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1-internal,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-4-11-1-internal-controller-metrics ClusterIP 10.108.39.186 10254/TCP 90m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1-internal,app.kubernetes.io/name=ingress-nginx service/nginx-service ClusterIP 10.99.244.61 80/TCP 85m app.kubernetes.io/name=proxy NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-4-11-1-controller 3/3 3 3 91m controller registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx deployment.apps/ingress-nginx-4-11-1-internal-controller 3/3 3 3 90m controller registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1-internal,app.kubernetes.io/name=ingress-nginx NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-4-11-1-controller-545b5f6457 3 3 3 89m controller registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx,pod-template-hash=545b5f6457 replicaset.apps/ingress-nginx-4-11-1-controller-5794bddc88 0 0 0 89m controller registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5794bddc88 replicaset.apps/ingress-nginx-4-11-1-controller-57cd6cdf8c 0 0 0 91m controller registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx,pod-template-hash=57cd6cdf8c replicaset.apps/ingress-nginx-4-11-1-internal-controller-594984bbdd 3 3 3 90m controller registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1-internal,app.kubernetes.io/name=ingress-nginx,pod-template-hash=594984bbdd NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/ingress-nginx-4-11-1-controller Deployment/ingress-nginx-4-11-1-controller memory: /90%, cpu: /90% 3 12 3 91m horizontalpodautoscaler.autoscaling/ingress-nginx-4-11-1-internal-controller Deployment/ingress-nginx-4-11-1-internal-controller memory: /90%, cpu: /90% 3 12 3 90m ```
pod resources ``` Name: ingress-nginx-4-11-1-controller-545b5f6457-428f5 Namespace: cloud-qa-f1 Priority: 0 Service Account: ingress-nginx-4-11-1 Node: minikube-m03/192.168.49.4 Start Time: Wed, 07 Aug 2024 14:35:42 +0300 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 pod-template-hash=545b5f6457 Annotations: kubectl.kubernetes.io/restartedAt: 2024-08-07T14:35:15+03:00 kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.2.16 IPs: IP: 10.244.2.16 Controlled By: ReplicaSet/ingress-nginx-4-11-1-controller-545b5f6457 Containers: controller: Container ID: docker://76fb8505db58e6561198a5d21fa6d15251b93e8583b0dabb9b2203864ed4cad5 Image: registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP, 6651/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-4-11-1-controller --election-id=ingress-controller-leader --controller-class=k8s.io/external-ingress-nginx --ingress-class=nginx-cloud-qa-f1 --configmap=$(POD_NAMESPACE)/ingress-nginx-4-11-1-controller --watch-namespace=$(POD_NAMESPACE) --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough=true State: Running Started: Wed, 07 Aug 2024 14:35:43 +0300 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2362232012800m Requests: cpu: 2 memory: 2362232012800m Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-4-11-1-controller-545b5f6457-428f5 (v1:metadata.name) POD_NAMESPACE: cloud-qa-f1 (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rdcvd (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-4-11-1-admission Optional: false kube-api-access-rdcvd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RELOAD 3m31s (x11 over 90m) nginx-ingress-controller NGINX reload triggered due to a change in configuration Name: ingress-nginx-4-11-1-controller-545b5f6457-bv9hq Namespace: cloud-qa-f1 Priority: 0 Service Account: ingress-nginx-4-11-1 Node: minikube/192.168.49.2 Start Time: Wed, 07 Aug 2024 14:35:53 +0300 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 pod-template-hash=545b5f6457 Annotations: kubectl.kubernetes.io/restartedAt: 2024-08-07T14:35:15+03:00 kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.0.12 IPs: IP: 10.244.0.12 Controlled By: ReplicaSet/ingress-nginx-4-11-1-controller-545b5f6457 Containers: controller: Container ID: docker://b2d73ebbdfab73811575c6dbb1aa07173dbdb44a1e0017e23c31afb16a6e3162 Image: registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP, 6651/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-4-11-1-controller --election-id=ingress-controller-leader --controller-class=k8s.io/external-ingress-nginx --ingress-class=nginx-cloud-qa-f1 --configmap=$(POD_NAMESPACE)/ingress-nginx-4-11-1-controller --watch-namespace=$(POD_NAMESPACE) --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough=true State: Running Started: Wed, 07 Aug 2024 14:35:53 +0300 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2362232012800m Requests: cpu: 2 memory: 2362232012800m Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-4-11-1-controller-545b5f6457-bv9hq (v1:metadata.name) POD_NAMESPACE: cloud-qa-f1 (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p7l2n (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-4-11-1-admission Optional: false kube-api-access-p7l2n: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RELOAD 3m31s (x11 over 90m) nginx-ingress-controller NGINX reload triggered due to a change in configuration Name: ingress-nginx-4-11-1-controller-545b5f6457-bxkjz Namespace: cloud-qa-f1 Priority: 0 Service Account: ingress-nginx-4-11-1 Node: minikube-m02/192.168.49.3 Start Time: Wed, 07 Aug 2024 14:35:42 +0300 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 pod-template-hash=545b5f6457 Annotations: kubectl.kubernetes.io/restartedAt: 2024-08-07T14:35:15+03:00 kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.1.14 IPs: IP: 10.244.1.14 Controlled By: ReplicaSet/ingress-nginx-4-11-1-controller-545b5f6457 Containers: controller: Container ID: docker://78746f22c6f7e1e06ade23eccb59787fb74981fb805b6e9e51a524b0eaec1da0 Image: registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP, 6651/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-4-11-1-controller --election-id=ingress-controller-leader --controller-class=k8s.io/external-ingress-nginx --ingress-class=nginx-cloud-qa-f1 --configmap=$(POD_NAMESPACE)/ingress-nginx-4-11-1-controller --watch-namespace=$(POD_NAMESPACE) --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough=true State: Running Started: Wed, 07 Aug 2024 14:35:43 +0300 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2362232012800m Requests: cpu: 2 memory: 2362232012800m Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-4-11-1-controller-545b5f6457-bxkjz (v1:metadata.name) POD_NAMESPACE: cloud-qa-f1 (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-98bcp (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-4-11-1-admission Optional: false kube-api-access-98bcp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RELOAD 3m31s (x11 over 90m) nginx-ingress-controller NGINX reload triggered due to a change in configuration Name: ingress-nginx-4-11-1-internal-controller-594984bbdd-6vkbl Namespace: cloud-qa-f1 Priority: 0 Service Account: ingress-nginx-4-11-1-internal Node: minikube-m03/192.168.49.4 Start Time: Wed, 07 Aug 2024 14:34:08 +0300 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1-internal app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 pod-template-hash=594984bbdd Annotations: kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.2.14 IPs: IP: 10.244.2.14 Controlled By: ReplicaSet/ingress-nginx-4-11-1-internal-controller-594984bbdd Containers: controller: Container ID: docker://65634630513d8421182071da336caf7451ed43ef445e940c3464145395274f51 Image: registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-4-11-1-internal-controller --election-id=internal-ingress-controller-leader --controller-class=k8s.io/internal-ingress-nginx --ingress-class=internal-nginx-cloud-qa-f1 --configmap=$(POD_NAMESPACE)/ingress-nginx-4-11-1-internal-controller --watch-namespace=$(POD_NAMESPACE) --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough=true State: Running Started: Wed, 07 Aug 2024 14:34:08 +0300 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2362232012800m Requests: cpu: 2 memory: 2362232012800m Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-4-11-1-internal-controller-594984bbdd-6vkbl (v1:metadata.name) POD_NAMESPACE: cloud-qa-f1 (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckpts (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-4-11-1-internal-admission Optional: false kube-api-access-ckpts: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RELOAD 3m31s (x6 over 92m) nginx-ingress-controller NGINX reload triggered due to a change in configuration Name: ingress-nginx-4-11-1-internal-controller-594984bbdd-f5xjk Namespace: cloud-qa-f1 Priority: 0 Service Account: ingress-nginx-4-11-1-internal Node: minikube/192.168.49.2 Start Time: Wed, 07 Aug 2024 14:34:08 +0300 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1-internal app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 pod-template-hash=594984bbdd Annotations: kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.0.10 IPs: IP: 10.244.0.10 Controlled By: ReplicaSet/ingress-nginx-4-11-1-internal-controller-594984bbdd Containers: controller: Container ID: docker://e4fedd1fc9105520f0a0fc7fd75099fb6eaea763a6c22e584d21345cbc61c6de Image: registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-4-11-1-internal-controller --election-id=internal-ingress-controller-leader --controller-class=k8s.io/internal-ingress-nginx --ingress-class=internal-nginx-cloud-qa-f1 --configmap=$(POD_NAMESPACE)/ingress-nginx-4-11-1-internal-controller --watch-namespace=$(POD_NAMESPACE) --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough=true State: Running Started: Wed, 07 Aug 2024 14:34:08 +0300 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2362232012800m Requests: cpu: 2 memory: 2362232012800m Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-4-11-1-internal-controller-594984bbdd-f5xjk (v1:metadata.name) POD_NAMESPACE: cloud-qa-f1 (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xx6zs (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-4-11-1-internal-admission Optional: false kube-api-access-xx6zs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RELOAD 3m31s (x6 over 92m) nginx-ingress-controller NGINX reload triggered due to a change in configuration Name: ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 Namespace: cloud-qa-f1 Priority: 0 Service Account: ingress-nginx-4-11-1-internal Node: minikube-m02/192.168.49.3 Start Time: Wed, 07 Aug 2024 14:33:53 +0300 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1-internal app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 pod-template-hash=594984bbdd Annotations: kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.1.12 IPs: IP: 10.244.1.12 Controlled By: ReplicaSet/ingress-nginx-4-11-1-internal-controller-594984bbdd Containers: controller: Container ID: docker://ac55d4ef3b1f5b433df062fd9d998aa26b172c87ea17e0796b8f3a7186afecc8 Image: registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-4-11-1-internal-controller --election-id=internal-ingress-controller-leader --controller-class=k8s.io/internal-ingress-nginx --ingress-class=internal-nginx-cloud-qa-f1 --configmap=$(POD_NAMESPACE)/ingress-nginx-4-11-1-internal-controller --watch-namespace=$(POD_NAMESPACE) --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough=true State: Running Started: Wed, 07 Aug 2024 14:33:53 +0300 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2362232012800m Requests: cpu: 2 memory: 2362232012800m Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-4-11-1-internal-controller-594984bbdd-gffj8 (v1:metadata.name) POD_NAMESPACE: cloud-qa-f1 (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vj6sp (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-4-11-1-internal-admission Optional: false kube-api-access-vj6sp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Guaranteed Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RELOAD 3m31s (x6 over 92m) nginx-ingress-controller NGINX reload triggered due to a change in configuration Name: nginx Namespace: cloud-qa-f1 Priority: 0 Service Account: default Node: minikube-m02/192.168.49.3 Start Time: Wed, 07 Aug 2024 14:39:22 +0300 Labels: app.kubernetes.io/name=proxy Annotations: kubedns-shepherd.io/dns-class-name: dnsclass-without-nameserver-sample Status: Running IP: 10.244.1.15 IPs: IP: 10.244.1.15 Containers: nginx: Container ID: docker://2beb4ec294ab78b4a9ba742d40fa80dc1768dde6e1ecf66ebdb559de455a47ef Image: nginx:stable Image ID: docker-pullable://nginx@sha256:3683b9aca292c84809ca60fb2c267a56d41879df91f6b0657694b992e1f3ee6e Port: 80/TCP Host Port: 0/TCP State: Running Started: Wed, 07 Aug 2024 14:39:33 +0300 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b8rfv (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-b8rfv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ```
svc resources ``` Name: ingress-nginx-4-11-1-controller Namespace: cloud-qa-f1 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1 app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 Annotations: meta.helm.sh/release-name: ingress-nginx-4-11-1 meta.helm.sh/release-namespace: cloud-qa-f1 Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.99.186.39 IPs: 10.99.186.39 LoadBalancer Ingress: 192.168.49.50 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32059/TCP Endpoints: 10.244.0.12:80,10.244.1.14:80,10.244.2.16:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 32554/TCP Endpoints: 10.244.0.12:443,10.244.1.14:443,10.244.2.16:443 Port: 6651-tcp 6651/TCP TargetPort: 6651-tcp/TCP NodePort: 6651-tcp 30647/TCP Endpoints: 10.244.0.12:6651,10.244.1.14:6651,10.244.2.16:6651 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 32480 Events: ``` ``` Name: ingress-nginx-4-11-1-internal-controller Namespace: cloud-qa-f1 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-4-11-1-internal app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.11.1 helm.sh/chart=ingress-nginx-4.11.1 Annotations: meta.helm.sh/release-name: ingress-nginx-4-11-1-internal meta.helm.sh/release-namespace: cloud-qa-f1 Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-4-11-1-internal,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.106.114.245 IPs: 10.106.114.245 LoadBalancer Ingress: 192.168.49.51 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32378/TCP Endpoints: 10.244.0.10:80,10.244.1.12:80,10.244.2.14:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30863/TCP Endpoints: 10.244.0.10:443,10.244.1.12:443,10.244.2.14:443 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 32723 Events: ```
Ingress resources ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx-cloud-qa-f1 name: test namespace: cloud-qa-f1 spec: ingressClassName: nginx-cloud-qa-f1 rules: - host: qa-f1.test.com http: paths: - backend: service: name: nginx-service port: number: 80 path: /nginx-service pathType: Prefix --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: internal-nginx-cloud-qa-f1 name: internal-test namespace: cloud-qa-f1 spec: ingressClassName: internal-nginx-cloud-qa-f1 rules: - host: internal-qa-f1.test.com http: paths: - backend: service: name: nginx-service port: number: 80 path: /nginx-service pathType: Prefix ```

How to reproduce this issue:

You can create a minikube cluster and apply the helm values and ingress object that I provided.

eminaktas commented 1 month ago

We followed this documentation to deploy: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

eminaktas commented 1 month ago

/cc @alperencelik

longwuyuan commented 1 month ago

Can you confirm that both the instances of the ingress-nginx controller have been made unique as described here https://kubernetes.github.io/ingress-nginx/faq/#multiple-controller-in-one-cluster

/remove-kind bug /kind support

eminaktas commented 1 month ago

Can you confirm that both the instances of the ingress-nginx controller have been made unique as described here https://kubernetes.github.io/ingress-nginx/faq/#multiple-controller-in-one-cluster

/remove-kind bug /kind support

Hi @longwuyuan, thanks for you quick response. Since I want to deploy them to in the same namespace I set all the values as described in the documentation. Nothing worked. I started to believe that we might be hitting a bug. You can also check the helm values that we applied.

longwuyuan commented 1 month ago

Hi @eminaktas , For analyzing the problem based on the logs and config, I was trying to look at these values ;

--set controller.electionID=nginx-two-leader \
--set controller.ingressClassResource.name=nginx-two \
--set controller.ingressClass=nginx-two \
--set controller.ingressClassResource.controllerValue="example.com/ingress-nginx-2" \
--set controller.ingressClassResource.enabled=true \
--set controller.ingressClassByName=true

for both instances of the controller.

But the info posted is too cumbursome to look at. For example the values file is so big and jam packed that its hard to quickly check only these values.

That leads to the second problem that the error message prints the name of a ingress resource and says it contains a invalid class. But there is no output of kubectl describe ing $ingressname or a cat $ingressfile.yaml .

If you can help out making the info more usable, maybe some kind of analysis will be easier. For example, the values file can only be limited to whatever you are customizing as the other keys will be configured from the default values file.

In any case you get the idea of how to debug this.

eminaktas commented 1 month ago

Hi, I provided the information as described in the ticket details. I agree that it is hard to dig into. Anyway, I think the below should help to show what we have done with the helm deployment.

internal controller:

--set controller.electionID=internal-ingress-controller-leader \
--set controller.ingressClassResource.name=internal-nginx-cloud-qa-f1 \
--set controller.ingressClass=internal-nginx-cloud-qa-f1 \
--set controller.ingressClassResource.controllerValue="k8s.io/internal-ingress-nginx" \
--set controller.ingressClassResource.enabled=true \
--set controller.ingressClassByName=true

external controller:

--set controller.electionID=ingress-controller-leader \
--set controller.ingressClassResource.name=nginx-cloud-qa-f1 \
--set controller.ingressClass=nginx-cloud-qa-f1 \
--set controller.ingressClassResource.controllerValue="k8s.io/external-ingress-nginx" \
--set controller.ingressClassResource.enabled=true \
--set controller.ingressClassByName=true

Here, I provided all the resources. Here is the output of the kubectl describe ing $ingressname

Name:             internal-test
Labels:           <none>
Namespace:        cloud-qa-f1
Address:          192.168.49.50
Ingress Class:    internal-nginx-cloud-qa-f1
Default backend:  <default>
Rules:
  Host                     Path  Backends
  ----                     ----  --------
  internal-qa-f1.test.com  
                           /nginx-service   nginx-service:80 (10.244.1.15:80)
Annotations:               kubernetes.io/ingress.class: internal-nginx-cloud-qa-f1
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    3m52s (x182 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x182 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x182 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x181 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x181 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x181 over 93m)  nginx-ingress-controller  Scheduled for sync

Name:             test
Labels:           <none>
Namespace:        cloud-qa-f1
Address:          192.168.49.50
Ingress Class:    nginx-cloud-qa-f1
Default backend:  <default>
Rules:
  Host            Path  Backends
  ----            ----  --------
  qa-f1.test.com  
                  /nginx-service   nginx-service:80 (10.244.1.15:80)
Annotations:      kubernetes.io/ingress.class: nginx-cloud-qa-f1
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    3m52s (x182 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x182 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x182 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x181 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x181 over 93m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    3m52s (x181 over 93m)  nginx-ingress-controller  Scheduled for sync

There is no customization only followed the documentation. I am willing to debug it but I am not familiar with the nginx ingress controller code, if you can point me I am willing to debug it.

longwuyuan commented 1 month ago

To state the obvious, get the output of kubectl get ing $ingressname -o yaml and compare the value for the ingress.spec.ingressClassName field , to explain the error message that you posted about the ingressClass being invalid ;

 "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/test" error="ingress d
oes not contain a valid IngressClass"
eminaktas commented 1 month ago

To state the obvious, get the output of kubectl get ing $ingressname -o yaml and compare the value for the ingress.spec.ingressClassName field , to explain the error message that you posted about the ingressClass being invalid ;

 "Ignoring ingress because of error while validating ingress class" ingress="cloud-qa-f1/test" error="ingress d
oes not contain a valid IngressClass"

I did it already. There is no typo or wrong definition. If you've checked it you'd also see it 👍

Here are the values of ingressClassNames internal-nginx-cloud-qa-f1 and nginx-cloud-qa-f1

 k get ingressclass
NAME                         CONTROLLER                      PARAMETERS   AGE
internal-nginx-cloud-qa-f1   k8s.io/internal-ingress-nginx   <none>       18h
nginx-cloud-qa-f1            k8s.io/external-ingress-nginx   <none>       18h

Those are valid values.

longwuyuan commented 1 month ago

can you share screen on meet.jit.si. I have looked at the info you posted and its possible that you are worrying about info level log messages. You have not posted a test in complete analyical details that shows that a curl is failing with unexpected response etc etc

eminaktas commented 1 month ago

can you share screen on meet.jit.si. I have looked at the info you posted and its possible that you are worrying about info level log messages. You have not posted a test in complete analyical details that shows that a curl is failing with unexpected response etc etc

I shared that log because it was the only thing I was able to notice to be honest. I think we can ignore that log I am not sure but it could be ValidatingWebhook because the scope is *. However, the configuration in nginx controller pods both include the both ingress resource definition which I believe should only include the referred one.

Of course I can, thanks for taking this seriously: https://meet.jit.si/ingress-nginx-debugging

longwuyuan commented 1 month ago

if this issue is resolved, please close the issue. thanks

eminaktas commented 1 month ago

@longwuyuan

Thank you for your time and support. I was able to find the issue after the meeting. We were setting the rbac.scope=true which prevents creating the ClusterRole (ingress-nginx/templates/clusterrole.yaml) and ClusterRoleBinding (ingress-nginx/templates/clusterrolebinding.yaml). When I set the configuration to false it got resolved and the issue hasn't repeated yet.

So, by default rbac.scope is set to false however, it is unclear for me why do we set it to false to create the resources?

longwuyuan commented 1 month ago

No idea which template consumes that key but the manifests we publish are explicit about the role/rolebinding related keys, so my guess is that a outside key is only existing to support user customization (that is highly and seriously not recommended)