The AGIC pod become not running

[JiayangZhou]

Describe the bug I have two AGIC pods deployed in a cluster and one of them is running fine (updating the appgw normally), but the other one is not in READY state (not updating the appgw), see the logs below and any ideas?

The AGIC is installed via helm, the specific version is application-gateway-kubernetes-ingress-staging/ingress-azure build number 32189.

To Reproduce Steps to reproduce the behavior:

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

  I0422 12:24:25.884393       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
  E0422 12:24:25.897591       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
  I0422 12:25:19.190168       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
  E0422 12:25:19.197227       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
  I0422 12:26:11.909281       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
  E0422 12:26:11.919917       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
  I0422 12:27:00.787450       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
  E0422 12:27:00.794544       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
  I0422 12:28:00.491178       1 reflector.go:255] Listing and watching *v1beta1.Ingress from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
  E0422 12:28:00.499707       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)

• Output of `kubectl logs .

  PS C:\Users\jzhou1> kubectl describe ingress grid-system-private-application-lb-integration-ingress -n m3dev02-env
  Name:             grid-system-private-application-lb-integration-ingress
  Namespace:        m3dev02-env
  Address:          100.112.50.199
  Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
  Rules:
  Host                                                   Path  Backends
  ----                                                   ----  --------
  m3dev02-integration.m3.azdevlocal.inforcloudsuite.com
                                                       /   grid-system-private-application-lb:37443 ()
  Annotations:                                             appgw.ingress.kubernetes.io/appgw-ssl-certificate: m3dev02-integration-m3-azdevlocal-inforcloudsuite-com
                                                       appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: m3dev02_cert
                                                       appgw.ingress.kubernetes.io/backend-hostname: m3dev02-integration.m3.azdevlocal.inforcloudsuite.com
                                                       appgw.ingress.kubernetes.io/backend-protocol: https
                                                       appgw.ingress.kubernetes.io/health-probe-hostname: m3dev02-integration.m3.azdevlocal.inforcloudsuite.com
                                                       appgw.ingress.kubernetes.io/health-probe-path: /grid/healthCheck
                                                       appgw.ingress.kubernetes.io/override-frontend-port: 443
                                                       appgw.ingress.kubernetes.io/request-timeout: 600
                                                       appgw.ingress.kubernetes.io/rewrite-rule-set: oauth_rewrite_ruleset
                                                       appgw.ingress.kubernetes.io/use-private-ip: true
                                                       kubernetes.io/ingress.class: application-gateway-private
                                                       meta.helm.sh/release-name: application-gateway-ingress-private
                                                       meta.helm.sh/release-namespace: m3dev02-env
  Events:                                                  <none>
  PS C:\Users\jzhou1> kubectl describe ingress grid-system-public-app-lb-saml-ingress -n m3dev02-env
  Name:             grid-system-public-app-lb-saml-ingress
  Namespace:        m3dev02-env
  Address:          20.80.208.232
  Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
  Rules:
  Host                                  Path  Backends
  ----                                  ----  --------
  m3dev02.m3.azdev.inforcloudsuite.com
                                      /   grid-system-public-app-lb:33443 ()
  Annotations:                            appgw.ingress.kubernetes.io/appgw-ssl-certificate: m3dev02-m3-azdev-inforcloudsuite-com
                                      appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: m3dev02_cert
                                      appgw.ingress.kubernetes.io/backend-hostname: m3dev02.m3.azdev.inforcloudsuite.com
                                      appgw.ingress.kubernetes.io/backend-protocol: https
                                      appgw.ingress.kubernetes.io/health-probe-hostname: m3dev02.m3.azdev.inforcloudsuite.com
                                      appgw.ingress.kubernetes.io/health-probe-path: /grid/healthCheck
                                      appgw.ingress.kubernetes.io/override-frontend-port: 443
                                      appgw.ingress.kubernetes.io/request-timeout: 600
                                      kubernetes.io/ingress.class: application-gateway-public
                                      meta.helm.sh/release-name: application-gateway-ingress-public
                                      meta.helm.sh/release-namespace: m3dev02-env
  Events:                                 <none>

• Any Azure support tickets associated with this issue.

[JiayangZhou]

The issue is resolved though by deleting the Not Ready AGIC pod, it became ready but how could I look into the cause to prevent it from happening in the future.

[severity1]

I am experiencing a variation of this issue. This is happening for me on kubernetes v1.23.3

I0428 00:25:37.765314       1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I0428 00:25:37.765335       1 httpserver.go:57] Starting API Server on :8123
I0428 00:25:48.492748       1 main.go:183] Ingress Controller will observe all namespaces.
I0428 00:25:48.613142       1 context.go:138] k8s context run started
I0428 00:25:48.613168       1 context.go:188] Waiting for initial cache sync
E0428 00:25:48.615665       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:25:48.617230       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:25:51.011040       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:25:55.937539       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:26:06.970103       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:26:31.776209       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:27:09.574870       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:28:04.555823       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:28:35.976707       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)
E0428 00:29:14.361793       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions)