This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
I'm not sure when this happens exactly... But it happens on a regular basis, and usually the cert-manager pods crash as well when it does. It's maybe linked to HTTPS cert renewal? Just a blind guess.
Note, I haven't tried yet upgrading to a newer release.
Ingress Controller details
Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
Name: ingress-appgw-deployment-7f95c87b96-l8tgp
Namespace: kube-system
Priority: 0
Service Account: ingress-appgw-sa
Node: aks-agentpool-18885632-vmss000008/10.192.1.184
Start Time: Fri, 04 Aug 2023 20:25:03 +0200
Labels: app=ingress-appgw
kubernetes.azure.com/managedby=aks
pod-template-hash=7f95c87b96
Annotations: checksum/config: 4ae721df21fa3243f1f0135ffe5c22d42d3e5bb03d715e55de87a1e4d8a9037f
cluster-autoscaler.kubernetes.io/safe-to-evict: true
kubernetes.azure.com/metrics-scrape: true
prometheus.io/path: /metrics
prometheus.io/port: 8123
prometheus.io/scrape: true
resource-id:
/subscriptions/145f2d93-d721-4d30-9784-042a706f137e/resourceGroups/rapmed-kc-v2/providers/Microsoft.ContainerService/managedClusters/rapme...
Status: Running
IP: 10.192.2.19
IPs:
IP: 10.192.2.19
Controlled By: ReplicaSet/ingress-appgw-deployment-7f95c87b96
Containers:
ingress-appgw-container:
Container ID: containerd://731fb33284bc39fd1ee11435548002fedba4666206631fbf0940292191030b6f
Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.3
Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:efd474df2c22af6e5abc201eb417098acec988feb70a763bb0782eda743fff42
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 31 Aug 2023 11:53:30 +0200
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Sun, 27 Aug 2023 15:10:13 +0200
Finished: Thu, 31 Aug 2023 11:53:19 +0200
Ready: True
Restart Count: 1
Limits:
cpu: 700m
memory: 600Mi
Requests:
cpu: 100m
memory: 20Mi
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
ingress-appgw-cm ConfigMap Optional: false
Environment:
AGIC_POD_NAMESPACE: kube-system (v1:metadata.namespace)
KUBERNETES_PORT_443_TCP_ADDR: rapmed-kc-v2-dns-bac4064f.hcp.germanywestcentral.azmk8s.io
KUBERNETES_PORT: tcp://rapmed-kc-v2-dns-bac4064f.hcp.germanywestcentral.azmk8s.io:443
KUBERNETES_PORT_443_TCP: tcp://rapmed-kc-v2-dns-bac4064f.hcp.germanywestcentral.azmk8s.io:443
KUBERNETES_SERVICE_HOST: rapmed-kc-v2-dns-bac4064f.hcp.germanywestcentral.azmk8s.io
AZURE_CLOUD_PROVIDER_LOCATION: /etc/kubernetes/azure.json
AGIC_POD_NAME: ingress-appgw-deployment-7f95c87b96-l8tgp (v1:metadata.name)
Mounts:
/etc/kubernetes/azure.json from cloud-provider-config (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h5czr (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
cloud-provider-config:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/azure.json
HostPathType: File
kube-api-access-h5czr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning NodeNotReady 36m node-controller Node is not ready
Warning Unhealthy 36m kubelet Readiness probe failed: Get "http://10.192.2.19:8123/health/ready": dial tcp 10.192.2.19:8123: connect: connection refused
Warning Failed 36m kubelet Error: failed to sync configmap cache: timed out waiting for the condition
Normal Created 36m (x2 over 3d21h) kubelet Created container ingress-appgw-container
Normal Pulled 36m (x2 over 36m) kubelet Container image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.3" already present on machine
Normal Started 36m (x2 over 3d21h) kubelet Started container ingress-appgw-container
Output of `kubectl logs .
I0831 09:52:52.816970 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync
I0831 09:52:53.231683 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync
I0831 09:52:53.610157 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync
W0831 09:53:18.659660 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: watch of *v1.Ingress ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0831 09:53:18.659665 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: watch of *v1.Endpoints ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0831 09:53:18.659690 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: watch of *v1.IngressClass ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0831 09:53:18.659709 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0831 09:53:18.659739 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0831 09:53:18.659763 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
I0831 09:53:19.684948 1 reflector.go:255] Listing and watching *v1.Secret from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
I0831 09:53:19.696648 1 reflector.go:255] Listing and watching *v1.Ingress from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
I0831 09:53:19.755422 1 reflector.go:255] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167
E0831 09:53:19.793053 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 123 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x16dcc60, 0x283a8b0})
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0008d9440})
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:48 +0x75
panic({0x16dcc60, 0x283a8b0})
/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/Azure/application-gateway-kubernetes-ingress/pkg/k8scontext.handlers.secretDelete({0x408080}, {0x1773e40, 0xc000341140})
/azure/pkg/k8scontext/secrets_handlers.go:73 +0x5f
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
/go/pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/controller.go:245
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/shared_informer.go:779 +0xdf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f2a185886c0)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000038738, {0x1afc440, 0xc00058e2d0}, 0x1, 0xc000346360)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0xc000038788)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc00014a900)
/go/pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/shared_informer.go:771 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:71 +0x88
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x14df09f]
goroutine 123 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0008d9440})
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x16dcc60, 0x283a8b0})
/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/Azure/application-gateway-kubernetes-ingress/pkg/k8scontext.handlers.secretDelete({0x408080}, {0x1773e40, 0xc000341140})
/azure/pkg/k8scontext/secrets_handlers.go:73 +0x5f
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
/go/pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/controller.go:245
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
/go/pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/shared_informer.go:779 +0xdf
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f2a185886c0)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000038738, {0x1afc440, 0xc00058e2d0}, 0x1, 0xc000346360)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0xc000038788)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc00014a900)
/go/pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/shared_informer.go:771 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/go/pkg/mod/k8s.io/apimachinery@v0.21.2/pkg/util/wait/wait.go:71 +0x88
Any Azure support tickets associated with this issue.
Describe the bug
The pod crashes.
Steps to reproduce the behavior:
I'm not sure when this happens exactly... But it happens on a regular basis, and usually the cert-manager pods crash as well when it does. It's maybe linked to HTTPS cert renewal? Just a blind guess.
Note, I haven't tried yet upgrading to a newer release.
Ingress Controller details
kubectl describe pod <ingress controller> . Thehelm list.