New custom error pages issue: Application Gateway Custom Error Status Code Is Invalid

[yevhenii0bozhenko]

Describe the bug Ingress controller is unable to update Application Gateway (e.g. new pods IP addresses, backend pools..) if there are (in preview) custom error pages configured (codes: 403, 502). The App Gateway feature is described here

I have custom error pages set on a Global Level (not on listeners one).

Ingress controller pods produce errors listed in kubectl logs section below.

Questions

• Is there any plans for these custom error codes to be supported by Kubernetes Ingress or any way to make it work with this case?
• What are the dependencies? Go modules or Azure APIs etc.

To Reproduce Steps to reproduce the behavior:

1. Set the custom error page on a Global Level according to this official instruction here
2. Restart your deployment that has ingress pointing to it
3. Check the ingress controller pod logs, Application Gateway Activity Log, Backend Pool configuration of the related deployment for errors.

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

  Name:             ingress-appgw-deployment-857b4bbd5d-wxkqr
  Namespace:        kube-system
  Priority:         0
  Service Account:  ingress-appgw-sa
  Node:             aks-default-XXX/10.1.20.122
  Start Time:       Wed, 16 Aug 2023 11:06:57 +0300
  Labels:           app=ingress-appgw
                kubernetes.azure.com/managedby=aks
                pod-template-hash=857b4bbd5d
  Annotations:      checksum/config: e0809d2c92ee39a97ffaeeb8829fd946c25f6fbb7fbba0e1d5d5911c69b0125d
                cluster-autoscaler.kubernetes.io/safe-to-evict: true
                kubernetes.azure.com/metrics-scrape: true
                prometheus.io/path: /metrics
                prometheus.io/port: 8123
                prometheus.io/scrape: true
                resource-id:
                  /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.ContainerService/managedClusters/b...
  Status:           Running
  IP:               10.1.20.137
  IPs:
  IP:           10.1.20.137
  Controlled By:  ReplicaSet/ingress-appgw-deployment-857b4bbd5d
  Containers:
  ingress-appgw-container:
  Container ID:   containerd://1fcea36af6ffef7424710f786570c772357a779041dc0c4e488920c387c9ec65
  Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.3
  Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:efd474df2c22af6e5abc201eb417098acec988feb70a763bb0782eda743fff42
  Port:           <none>
  Host Port:      <none>
  State:          Running
    Started:      Wed, 16 Aug 2023 11:07:07 +0300
  Ready:          True
  Restart Count:  0
  Limits:
    cpu:     700m
    memory:  600Mi
  Requests:
    cpu:      100m
    memory:   20Mi
  Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
  Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
  Environment Variables from:
    ingress-appgw-cm  ConfigMap  Optional: false
  Environment:
    AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
    KUBERNETES_PORT_443_TCP_ADDR:   XXX.azmk8s.io
    KUBERNETES_PORT:                tcp://XXX.azmk8s.io:443
    KUBERNETES_PORT_443_TCP:        tcp://XXX.azmk8s.io:443
    KUBERNETES_SERVICE_HOST:        XXX.azmk8s.io
    AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
    AGIC_POD_NAME:                  ingress-appgw-deployment-857b4bbd5d-wxkqr (v1:metadata.name)
  Mounts:
    /etc/kubernetes/azure.json from cloud-provider-config (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t2kqr (ro)
  Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
  Volumes:
  cloud-provider-config:
  Type:          HostPath (bare host directory volume)
  Path:          /etc/kubernetes/azure.json
  HostPathType:  File
  kube-api-access-t2kqr:
  Type:                    Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds:  3607
  ConfigMapName:           kube-root-ca.crt
  ConfigMapOptional:       <nil>
  DownwardAPI:             true
  QoS Class:                   Burstable
  Node-Selectors:              <none>
  Tolerations:                 CriticalAddonsOnly op=Exists
                           node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                           node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                           node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:                      <none>

• Output of `kubectl logs .

  I1106 08:30:55.003540       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
  I1106 08:30:56.454097       1 mutate_app_gateway.go:183] END AppGateway deployment
  E1106 08:30:56.454179       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayCustomErrorStatusCodeIsInvalid" Message="Custom error page has an invalid HTTP status code '500', for applicationGateways /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX. Valid status codes are: '[403, 502]'." Details=[]
  E1106 08:30:56.454192       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayCustomErrorStatusCodeIsInvalid" Message="Custom error page has an invalid HTTP status code '500', for applicationGateways /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX. Valid status codes are: '[403, 502]'." Details=[]
  I1106 08:30:56.454315       1 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"ingress-appgw-deployment-857b4bbd5d-wxkqr", UID:"XXX", APIVersion:"v1", ResourceVersion:"140294170", FieldPath:""}): type: 'Warning' reason: 'FailedApplyingAppGwConfig' network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayCustomErrorStatusCodeIsInvalid" Message="Custom error page has an invalid HTTP status code '500', for applicationGateways /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX. Valid status codes are: '[403, 502]'." Details=[]

• Any Azure support tickets associated with this issue.

This issue is related to issue #733 and PR #1350 but exposes another use case.

[akshaysngupta]

@yevhenii0bozhenko I suspect that this is happening because AGIC is using older azure sdk.