dns-controller-manager reports invalid DNS entry for 5-10 mins after upgrade

krimeshshah commented 9 months ago

What happened: After we upgrade nginx ingress controller recent version 10.3.0 from 10.1.0, we observed that dns controller manager reported error "Normal dns-annotation dns-controller-manager : dns entry invalid: no target or text specified" for short duration and it fixed itself in sometime. We also observed during that time if we execute command kubectl get ingress -w , the addressfield was empty for short duration and then it got filled with address it self.

What you expected to happen: There should not be missing address from ingress resource and the above mentioned error should not appear.

its unclear but somehow , spec.loadbalancer.ingress filed got emptied somehow.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): Release: 1.9.5, nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

Environment:

Cloud provider or hardware configuration: aws
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
- Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
Basic cluster related info:
- kubectl version v1.21.6 kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
  
  ip-10-250-145-204.eu-central-1.compute.internal Ready 5h9m v1.27.8 10.250.145.204
  Garden Linux 1312.1 6.1.63-gardenlinux-cloud-amd64 containerd://1.6.24 ip-10-250-67-178.eu-central-1.compute.internal Ready 100m v1.27.8 10.250.67.178 Garden Linux 1312.1 6.1.63-gardenlinux-cloud-amd64 containerd://1.6.24 ip-10-250-88-33.eu-central-1.compute.internal Ready 5h9m v1.27.8 10.250.88.33 Garden Linux 1312.1 6.1.63-gardenlinux-cloud-amd64 containerd://1.6.24
How was the ingress-nginx-controller installed: If helm was used then please show output of helm ls -A | grep -i ingress
nginx-ingress-controller-experimental-lp nginx 3 2024-02-22 12:31:33.175062855 +0530 IST deployed nginx-ingress-controller-0.42.0 7.6.6
- If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

 global:
  imagePullSecrets:
  - artifactory
  imageRegistry: remote-docker.common.repositories.cloud.sap
namespace: nginx
namespaced: false
nginx-ingress-controller:
  autoscaling:
    enabled: true
    maxReplicas: 6
    minReplicas: 3
  config:
    allow-snippet-annotations: "true"
    keep-alive: "305"
    use-proxy-protocol: "true"
  defaultBackend:
    networkPolicy:
      enabled: false
    podAntiAffinityPreset: hard
    resources:
      limits:
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 256Mi
  ingressClassResource:
    default: true
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
  pdb:
    create: true
    minAvailable: 1
  podAntiAffinityPreset: hard
  publishService:
    enabled: true
  replicaCount: 2
  resources:
    limits:
      memory: 512Mi
    requests:
      cpu: 250m
      memory: 300Mi
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3000"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-target-node-labels: ingress
    externalTrafficPolicy: Cluster
    networkPolicy:
      enabled: false
    ports:
      http: false
  watchIngressWithoutClass: true

If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

Current State of the controller:
- kubectl describe ingressclasses
- kubectl -n <ingresscontrollernamespace> get all -A -o wide
- kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:         nginx-ingress-controller-int-lp-76d66bd978-nj9w2
Namespace:    nginx
Priority:     0
Node:         ip-10-250-11-28.eu-central-1.compute.internal/10.250.11.28
Start Time:   Mon, 04 Mar 2024 09:17:25 +0530
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx-ingress-controller-int-lp
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=nginx-ingress-controller
              app.kubernetes.io/version=1.9.6
              helm.sh/chart=nginx-ingress-controller-10.5.2
              pod-template-hash=76d66bd978
Annotations:  cni.projectcalico.org/containerID: 9f6f05ea78d9ae4387de3e8717203a086fe42d8c2cd9f9a1c05d471ea70e992b
              cni.projectcalico.org/podIP: 100.96.1.100/32
              cni.projectcalico.org/podIPs: 100.96.1.100/32
Status:       Running
IP:           100.96.1.100
IPs:
  IP:           100.96.1.100
Controlled By:  ReplicaSet/nginx-ingress-controller-int-lp-76d66bd978
Containers:
  controller:
    Container ID:  containerd://70f16004dc14ae3050a2dd9723049cc80d37264b2d2a4132a9a2ccfc65860f8f
    Image:         remote-docker.common.repositories.cloud.sap/bitnami/nginx-ingress-controller:1.9.6-debian-12-r8
    Image ID:      remote-docker.common.repositories.cloud.sap/bitnami/nginx-ingress-controller@sha256:9b34cde2ca85a87d0c3ac8e8d0e79977670ddcd63720f7fa801749161573df0a
    Ports:         8080/TCP, 8443/TCP, 10254/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      --default-backend-service=$(POD_NAMESPACE)/nginx-ingress-controller-int-lp-default-backend
      --publish-service=nginx/nginx-ingress-controller-int-lp
      --http-port=8080
      --https-port=8443
      --healthz-port=10254
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --configmap=$(POD_NAMESPACE)/nginx-ingress-controller-int-lp
      --watch-ingress-without-class=true
    State:          Running
      Started:      Mon, 04 Mar 2024 09:17:32 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  512Mi
    Requests:
      cpu:      250m
      memory:   300Mi
    Liveness:   http-get http://:metrics/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:metrics/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:                 nginx-ingress-controller-int-lp-76d66bd978-nj9w2 (v1:metadata.name)
      POD_NAMESPACE:            nginx (v1:metadata.namespace)
      KUBERNETES_SERVICE_HOST:  api.int.lp.internal.live.k8s.ondemand.com
    Mounts:
      /etc/ingress-controller/telemetry from telemetry (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4xwc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  telemetry:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-f4xwc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     nginx-ingress-controller-int-lp
Namespace:                nginx
Labels:                   app.kubernetes.io/component=controller
                      app.kubernetes.io/instance=nginx-ingress-controller-int-lp
                      app.kubernetes.io/managed-by=Helm
                      app.kubernetes.io/name=nginx-ingress-controller
                      app.kubernetes.io/version=1.9.6
                      helm.sh/chart=nginx-ingress-controller-10.5.2
Annotations:              argocd.argoproj.io/tracking-id: nginx-ingress-controller-int-lp:/Service:nginx/nginx-ingress-controller-int-lp
                      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 3000
                      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
                      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: *
                      service.beta.kubernetes.io/aws-load-balancer-target-node-labels: ingress
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress-controller-int-lp,app.kubernetes.io/name=nginx-ingress-controller
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       100.65.239.161
IPs:                      100.65.239.161
LoadBalancer Ingress:     a869dcd7800204ae09ed103542d6be67-1846820854.eu-central-1.elb.amazonaws.com
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32753/TCP
Endpoints:                100.96.1.100:8443,100.96.4.156:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
Type    Reason               Age                 From                Message
----    ------               ----                ----                -------
Normal  UpdatedLoadBalancer  27m (x2 over 119m)  service-controller  Updated load balancer with new hosts

Current state of ingress object, if applicable:

kubectl -n <appnamespace> get all,ing -o wide


NAME                                                                  READY   STATUS    RESTARTS   AGE    IP              NODE                                             NOMINATED NODE   READINESS GATES
pod/alertmanager-prometheus-stack-alertmanager-0                      2/2     Running   0          28h    100.96.4.162    ip-10-250-84-222.eu-central-1.compute.internal   <none>           <none>
pod/monitoring-int-lp-grafana-58d99c7d9b-jv7lw                        4/4     Running   0          13d    100.96.3.189    ip-10-250-6-201.eu-central-1.compute.internal    <none>           <none>
pod/monitoring-int-lp-kube-state-metrics-68c9bfcc77-jq75b             1/1     Running   0          28h    100.96.4.161    ip-10-250-84-222.eu-central-1.compute.internal   <none>           <none>
pod/monitoring-int-lp-prometheus-adapter-5cf45bf76-nzhxk              1/1     Running   0          13d    100.96.4.12     ip-10-250-84-222.eu-central-1.compute.internal   <none>           <none>
pod/monitoring-int-lp-prometheus-blackbox-exporter-5979844d89-52rfr   1/1     Running   0          115m   100.96.4.153    ip-10-250-84-222.eu-central-1.compute.internal   <none>           <none>
pod/monitoring-int-lp-prometheus-node-exporter-2v9zc                  1/1     Running   0          28h    10.250.11.28    ip-10-250-11-28.eu-central-1.compute.internal    <none>           <none>
pod/monitoring-int-lp-prometheus-node-exporter-bt97n                  1/1     Running   0          23m    10.250.74.14    ip-10-250-74-14.eu-central-1.compute.internal    <none>           <none>
pod/monitoring-int-lp-prometheus-node-exporter-d44nd                  1/1     Running   0          28h    10.250.27.189   ip-10-250-27-189.eu-central-1.compute.internal   <none>           <none>
pod/monitoring-int-lp-prometheus-node-exporter-k6p96                  1/1     Running   0          28h    10.250.84.222   ip-10-250-84-222.eu-central-1.compute.internal   <none>           <none>
pod/monitoring-int-lp-prometheus-node-exporter-zpb2p                  1/1     Running   0          28h    10.250.6.201    ip-10-250-6-201.eu-central-1.compute.internal    <none>           <none>
pod/prometheus-prometheus-stack-prometheus-0                          2/2     Running   0          115m   100.96.4.163    ip-10-250-84-222.eu-central-1.compute.internal   <none>           <none>
pod/prometheus-stack-operator-f464cb676-4fg2t                         1/1     Running   0          115m   100.96.0.54     ip-10-250-27-189.eu-central-1.compute.internal   <none>           <none>

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 3y176d app.kubernetes.io/name=alertmanager service/monitoring-int-lp-grafana ClusterIP 100.65.11.129 80/TCP 3y182d app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana service/monitoring-int-lp-kube-state-metrics ClusterIP 100.68.5.222 8080/TCP 3y182d app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics service/monitoring-int-lp-prometheus-adapter ClusterIP 100.70.191.131 443/TCP 3y182d app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter service/monitoring-int-lp-prometheus-blackbox-exporter ClusterIP 100.64.90.243 9115/TCP 2y148d app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter service/monitoring-int-lp-prometheus-node-exporter ClusterIP 100.68.46.204 9100/TCP 3y182d app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-node-exporter service/prometheus-operated ClusterIP None 9090/TCP 3y176d app.kubernetes.io/name=prometheus service/prometheus-stack-alertmanager ClusterIP 100.68.197.208 9093/TCP,8080/TCP 3y176d alertmanager=prometheus-stack-alertmanager,app.kubernetes.io/name=alertmanager service/prometheus-stack-operator ClusterIP 100.70.55.111 8080/TCP 3y176d app=kube-prometheus-stack-operator,release=monitoring-int-lp service/prometheus-stack-prometheus ClusterIP 100.64.49.106 9090/TCP,8080/TCP 3y176d app.kubernetes.io/name=prometheus,operator.prometheus.io/name=prometheus-stack-prometheus

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR daemonset.apps/monitoring-int-lp-prometheus-node-exporter 5 5 5 5 5 kubernetes.io/os=linux 468d node-exporter external-releases-quay-proxy.common.repositories.cloud.sap/prometheus/node-exporter:v1.7.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-node-exporter

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/monitoring-int-lp-grafana 1/1 1 1 3y157d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana deployment.apps/monitoring-int-lp-kube-state-metrics 1/1 1 1 468d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics deployment.apps/monitoring-int-lp-prometheus-adapter 1/1 1 1 2y135d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter deployment.apps/monitoring-int-lp-prometheus-blackbox-exporter 1/1 1 1 2y148d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter deployment.apps/prometheus-stack-operator 1/1 1 1 3y176d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.71.2 app=kube-prometheus-stack-operator,release=monitoring-int-lp

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/monitoring-int-lp-grafana-58d99c7d9b 1 1 1 33d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=58d99c7d9b replicaset.apps/monitoring-int-lp-grafana-699cdfd7b 0 0 0 73d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=699cdfd7b replicaset.apps/monitoring-int-lp-grafana-789c44d656 0 0 0 157d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=789c44d656 replicaset.apps/monitoring-int-lp-grafana-7c765f6794 0 0 0 162d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=7c765f6794 replicaset.apps/monitoring-int-lp-grafana-7cfcb656fd 0 0 0 182d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=7cfcb656fd replicaset.apps/monitoring-int-lp-grafana-7d7cc846f9 0 0 0 56d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=7d7cc846f9 replicaset.apps/monitoring-int-lp-grafana-7fd6d5dbd 0 0 0 57d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=7fd6d5dbd replicaset.apps/monitoring-int-lp-grafana-84bb44f76d 0 0 0 133d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=84bb44f76d replicaset.apps/monitoring-int-lp-grafana-8675598c65 0 0 0 85d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=8675598c65 replicaset.apps/monitoring-int-lp-grafana-bf875f8d8 0 0 0 134d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.21.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=bf875f8d8 replicaset.apps/monitoring-int-lp-grafana-cf4469b56 0 0 0 91d grafana-sc-dashboard,grafana-sc-datasources,grafana-sc-notifiers,grafana remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/kiwigrid/k8s-sidecar:1.25.0,remote-docker.common.cdn.repositories.cloud.sap/grafana/grafana:7.5.17 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=grafana,pod-template-hash=cf4469b56 replicaset.apps/monitoring-int-lp-kube-state-metrics-579f578549 0 0 0 308d kube-state-metrics k8s-gcr-io.common.cdn.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.8.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=579f578549 replicaset.apps/monitoring-int-lp-kube-state-metrics-5bbbc66b4d 0 0 0 239d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.9.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=5bbbc66b4d replicaset.apps/monitoring-int-lp-kube-state-metrics-68c9bfcc77 1 1 1 28h kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=68c9bfcc77 replicaset.apps/monitoring-int-lp-kube-state-metrics-68cb6fc6b4 0 0 0 211d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.9.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=68cb6fc6b4 replicaset.apps/monitoring-int-lp-kube-state-metrics-794677b84d 0 0 0 91d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=794677b84d replicaset.apps/monitoring-int-lp-kube-state-metrics-7bf8df8bcd 0 0 0 162d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=7bf8df8bcd replicaset.apps/monitoring-int-lp-kube-state-metrics-7cb4c45895 0 0 0 134d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=7cb4c45895 replicaset.apps/monitoring-int-lp-kube-state-metrics-7cbbf76fbc 0 0 0 56d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=7cbbf76fbc replicaset.apps/monitoring-int-lp-kube-state-metrics-8445767bbf 0 0 0 263d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.9.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=8445767bbf replicaset.apps/monitoring-int-lp-kube-state-metrics-9d65956cf 0 0 0 42d kube-state-metrics registry-k8s-io.common.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.10.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=9d65956cf replicaset.apps/monitoring-int-lp-kube-state-metrics-c4bb86564 0 0 0 281d kube-state-metrics k8s-gcr-io.common.cdn.repositories.cloud.sap/kube-state-metrics/kube-state-metrics:v2.8.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=c4bb86564 replicaset.apps/monitoring-int-lp-prometheus-adapter-5cb7b5bfcd 0 0 0 308d prometheus-adapter k8s-gcr-io.common.cdn.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.10.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=5cb7b5bfcd replicaset.apps/monitoring-int-lp-prometheus-adapter-5cf45bf76 1 1 1 56d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=5cf45bf76 replicaset.apps/monitoring-int-lp-prometheus-adapter-64f6d69bf5 0 0 0 340d prometheus-adapter k8s-gcr-io.common.cdn.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.10.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=64f6d69bf5 replicaset.apps/monitoring-int-lp-prometheus-adapter-668fd9f84d 0 0 0 134d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=668fd9f84d replicaset.apps/monitoring-int-lp-prometheus-adapter-694cd4cf7d 0 0 0 212d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.10.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=694cd4cf7d replicaset.apps/monitoring-int-lp-prometheus-adapter-6bdfb87b56 0 0 0 183d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=6bdfb87b56 replicaset.apps/monitoring-int-lp-prometheus-adapter-7654bf57cb 0 0 0 211d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=7654bf57cb replicaset.apps/monitoring-int-lp-prometheus-adapter-7b5d756cc6 0 0 0 197d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=7b5d756cc6 replicaset.apps/monitoring-int-lp-prometheus-adapter-7bb949dc4c 0 0 0 84d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.2 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=7bb949dc4c replicaset.apps/monitoring-int-lp-prometheus-adapter-7c5d79d85d 0 0 0 148d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=7c5d79d85d replicaset.apps/monitoring-int-lp-prometheus-adapter-c9479758f 0 0 0 162d prometheus-adapter registry-k8s-io.common.repositories.cloud.sap/prometheus-adapter/prometheus-adapter:v0.11.1 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-adapter,pod-template-hash=c9479758f replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-58f7d78c49 0 0 0 239d blackbox-exporter prom/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=58f7d78c49 replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-5979844d89 1 1 1 29d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=5979844d89 replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-6cc76cdb75 0 0 0 147d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=6cc76cdb75 replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-7c9b8b5cf6 0 0 0 351d blackbox-exporter prom/blackbox-exporter:v0.23.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=7c9b8b5cf6 replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-7cf455b95d 0 0 0 43d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=7cf455b95d replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-7f8966f4d 0 0 0 225d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=7f8966f4d replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-84d866bf88 0 0 0 99d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=84d866bf88 replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-85bcc58f84 0 0 0 294d blackbox-exporter prom/blackbox-exporter:v0.23.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=85bcc58f84 replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-866b6dffcf 0 0 0 337d blackbox-exporter prom/blackbox-exporter:v0.23.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=866b6dffcf replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-9fb6bf4fc 0 0 0 197d blackbox-exporter quay.io/prometheus/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=9fb6bf4fc replicaset.apps/monitoring-int-lp-prometheus-blackbox-exporter-d6f4c75d9 0 0 0 267d blackbox-exporter prom/blackbox-exporter:v0.24.0 app.kubernetes.io/instance=monitoring-int-lp,app.kubernetes.io/name=prometheus-blackbox-exporter,pod-template-hash=d6f4c75d9 replicaset.apps/prometheus-stack-operator-55b98dd4c7 0 0 0 56d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.70.0 app=kube-prometheus-stack-operator,pod-template-hash=55b98dd4c7,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-64f5dbb978 0 0 0 73d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.69.1 app=kube-prometheus-stack-operator,pod-template-hash=64f5dbb978,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-66c5c8c674 0 0 0 15d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.71.2 app=kube-prometheus-stack-operator,pod-template-hash=66c5c8c674,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-67dd568745 0 0 0 134d kube-prometheus-stack external-releases-quay-proxy.common.cdn.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.68.0 app=kube-prometheus-stack-operator,pod-template-hash=67dd568745,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-6bccfcccdd 0 0 0 91d kube-prometheus-stack external-releases-quay-proxy.common.cdn.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.69.1 app=kube-prometheus-stack-operator,pod-template-hash=6bccfcccdd,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-76f44545f6 0 0 0 120d kube-prometheus-stack external-releases-quay-proxy.common.cdn.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.68.0 app=kube-prometheus-stack-operator,pod-template-hash=76f44545f6,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-776cd5495c 0 0 0 42d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.71.0 app=kube-prometheus-stack-operator,pod-template-hash=776cd5495c,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-77757bf4d4 0 0 0 57d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.69.1 app=kube-prometheus-stack-operator,pod-template-hash=77757bf4d4,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-8b9bc7cf9 0 0 0 42d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.70.0 app=kube-prometheus-stack-operator,pod-template-hash=8b9bc7cf9,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-8c4857bd8 0 0 0 29d kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.71.2 app=kube-prometheus-stack-operator,pod-template-hash=8c4857bd8,release=monitoring-int-lp replicaset.apps/prometheus-stack-operator-f464cb676 1 1 1 28h kube-prometheus-stack external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-operator:v0.71.2 app=kube-prometheus-stack-operator,pod-template-hash=f464cb676,release=monitoring-int-lp

NAME READY AGE CONTAINERS IMAGES statefulset.apps/alertmanager-prometheus-stack-alertmanager 1/1 532d alertmanager,config-reloader external-releases-quay-proxy.common.repositories.cloud.sap/prometheus/alertmanager:v0.27.0,external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-config-reloader:v0.71.2 statefulset.apps/prometheus-prometheus-stack-prometheus 1/1 166d prometheus,config-reloader external-releases-quay-proxy.common.repositories.cloud.sap/prometheus/prometheus:v2.50.1,external-releases-quay-proxy.common.repositories.cloud.sap/prometheus-operator/prometheus-config-reloader:v0.71.2

NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/monitoring-int-lp-grafana grafana.ingress.int.lp.shoot.live.k8s-hana.ondemand.com a869dcd7800204ae09ed103542d6be67-1846820854.eu-central-1.elb.amazonaws.com 80, 443 3y182d

  - `kubectl -n <appnamespace> describe ing <ingressname>`

Name: monitoring-int-lp-grafana Namespace: monitoring Address: a869dcd7800204ae09ed103542d6be67-1846820854.eu-central-1.elb.amazonaws.com Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) TLS: grafana-tls terminates grafana.ingress.int.lp.shoot.live.k8s-hana.ondemand.com Rules: Host Path Backends

grafana.ingress.int.lp.shoot.live.k8s-hana.ondemand.com
/ monitoring-int-lp-grafana:80 (100.96.3.189:3000) Annotations: argocd.argoproj.io/sync-options: Prune=false, Delete=false argocd.argoproj.io/tracking-id: monitoring-int-lp:networking.k8s.io/Ingress:monitoring/monitoring-int-lp-grafana

Events:

If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
- kubectl describe ... of any custom configmap(s) created and in use
- Any other related information that may help

How to reproduce this issue:

You will see the address files empty after some time for a short duration and then it will itself restore the address field with DNS entry Open one more tab in the terminal and execute kubectl describe ingress -n --> You should see Normal dns-annotation dns-controller-manager : dns entry invalid: no target or text specified error This error only stays for a short time say 5-10 mins and itself fixes but this 5-10 mins causes downtime. As minimally and precisely as possible. Keep in mind we do not have access to your cluster or application. Help up us (if possible) reproducing the issue using minikube or kind. ## Install minikube/kind - Minikube https://minikube.sigs.k8s.io/docs/start/ - Kind https://kind.sigs.k8s.io/docs/user/quick-start/ ## Install the ingress controller kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml ## Install an application that will act as default backend (is just an echo app) kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml ## Create an ingress (please add any additional annotation required) echo " apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: foo-bar annotations: kubernetes.io/ingress.class: nginx spec: ingressClassName: nginx # omit this if you're on controller version below 1.0.0 rules: - host: foo.bar http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 " | kubectl apply -f - ## make a request POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME) kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' localhost --->

Anything else we need to know:

k8s-ci-robot commented 9 months ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

longwuyuan commented 9 months ago

/remove-kind bug

this does not look like a release from this project
```
recent version 10.3.0 from 10.1.0,
```

this also does not look like release from this project

nginx-ingress-controller-experimental-lp nginx 3 2024-02-22 12:31:33.175062855 +0530 IST deployed nginx-ingress-controller-0.42.0 7.6.6

this does seem related to this project

Release: 1.9.5, nginx version: nginx/1.21.6

You have skipped all important questions in the template. Answer them so that a reader here can know more and help you
What is the output of kubectl -n <controllernamespace> get po -o yaml | grep -i image

/triage needs-information

longwuyuan commented 9 months ago

The information you have posted is not in markdown format and hence hard to read. You can help by editing the issue description and do both ;

edit current info to confirm to markdown format
post answers to unanswered questions also in markdown format

krimeshshah commented 9 months ago

Now, I have added the mark down format and also provided answers to possible important questions.

strongjz commented 8 months ago

This looks like the bitnami chart for ingress-nginx, not the project one, and your kubectl version says v1.21.6, what version of a k8s cluster are you testing this on? We are only testing and supporting back to 1.26 currently.

github-actions[bot] commented 7 months ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

kubernetes / ingress-nginx

dns-controller-manager reports invalid DNS entry for 5-10 mins after upgrade #11009