502 error with all but one route in path-based routing

Describe the bug I have the routes of type prefix defined in my ingress:

'/api/users/signin'
'/'

The '/' path lead to a client webapp that is successfully being loaded when I type the website address. When I try to signin via the 2nd route , I get a 502 error.

To Reproduce

Kubernetes apply ingress script

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
  tls:
    - hosts:
        - mydomain.com
      secretName: mysecret-tls
  rules:
    - host: mydomain.com
      http:
        paths:
          - pathType: Prefix
            path: /api/users/signin
            backend:
              service:
                name: auth-srv
                port:
                  number: 3000
          - pathType: Prefix
            path: /
            backend:
              service:
                name: client-srv
                port:
                  number: 3000

Ingress Controller details

Output of kubectl describe pod <ingress controller>

Name:             ingress-appgw-deployment-8557f66847-d4pt4
Namespace:        kube-system
Priority:         0
Service Account:  ingress-appgw-sa
Node:             aks-agentpool-40935132-vmss000001/10.0.0.222
Start Time:       Thu, 06 Apr 2023 06:38:12 +0200
Labels:           app=ingress-appgw
              kubernetes.azure.com/managedby=aks
              pod-template-hash=8557f66847
Annotations:      checksum/config: 00bac1c74711db7e62e7de4a4701f1ebdad136bb03e29ba9638f263f48509564
              cluster-autoscaler.kubernetes.io/safe-to-evict: true
              kubernetes.azure.com/metrics-scrape: true
              prometheus.io/path: /metrics
              prometheus.io/port: 8123
              prometheus.io/scrape: true
              resource-id:
                /subscriptions/abcd/resourceGroups/RG/providers/Microsoft.ContainerService/managedClusters/...
Status:           Running
IP:               10.0.0.123
IPs:
IP:           10.0.0.123
Controlled By:  ReplicaSet/ingress-appgw-deployment-8557f66847
Containers:
ingress-appgw-container:
Container ID:   containerd://1ebf27c6297035916f07c7a466e5a946cf773f8fce622071b49607c380a8955d
Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.3
Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:efd474df2c22af6e5abc201eb417098acec988feb70a763bb0782eda743fff42
Port:           <none>
Host Port:      <none>
State:          Running
  Started:      Thu, 06 Apr 2023 06:38:20 +0200
Ready:          True
Restart Count:  0
Limits:
  cpu:     700m
  memory:  600Mi
Requests:
  cpu:      100m
  memory:   20Mi
Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
  ingress-appgw-cm  ConfigMap  Optional: false
Environment:
  KUBERNETES_PORT_443_TCP_ADDR:   myapp-dns-irb4tgur.hcp.germanywestcentral.azmk8s.io
  KUBERNETES_PORT:                tcp://myapp-dns-irb4tgur.hcp.germanywestcentral.azmk8s.io:443
  KUBERNETES_PORT_443_TCP:        tcp://myapp-dns-irb4tgur.hcp.germanywestcentral.azmk8s.io:443
  KUBERNETES_SERVICE_HOST:        myapp-dns-irb4tgur.hcp.germanywestcentral.azmk8s.io
  AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
  AGIC_POD_NAME:                  ingress-appgw-deployment-8557f66847-d4pt4 (v1:metadata.name)
  AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
Mounts:
  /etc/kubernetes/azure.json from cloud-provider-config (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tz8hg (ro)
Conditions:
Type              Status
Initialized       True
Ready             True
ContainersReady   True
PodScheduled      True
Volumes:
cloud-provider-config:
Type:          HostPath (bare host directory volume)
Path:          /etc/kubernetes/azure.json
HostPathType:  File
kube-api-access-tz8hg:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  3607
ConfigMapName:           kube-root-ca.crt
ConfigMapOptional:       <nil>
DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 :NoExecute op=Exists
                         :NoSchedule op=Exists
                         CriticalAddonsOnly op=Exists
Events:                      <none>

Output of kubectl logs <ingress controller>. The logs show nothing butI0330 07:38:33.413508 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync`

When looking at the app gateway logs in Azure portal, I can see that for a requested route the correct backend setting name and backend pool name are selected for all routes (in columns backendPoolName_s and backendSettingName_s). What is strange, the serverRouted_s column lists for the successful request to path '/' the correct IP: 10.0.0.234:3000 However, for the unsuccessful request to signin, the serverRouted_s column shows the name of the backendPoolName instead of the resolved IP address. Also, the server response latency for the not working signin request is 0.

Looking at the backend settings and backend pool settings in Azure portal, there is no obvious difference between the working client and the not working signin configurations. The backend target in the rules menu also look good. If I choose the signin route as the backend target in Azure portal (=the target that the request is routed too when no path matches), the error still occurs. When I remove the '/' route entirely and only leave the signin route and try to send a request with Postman, the same 502 error persists.

Not sure whether this not resolved serverRouted_s entry in the debug log points to the true cause of the error. I am very greatful for any hint where to check because it all looks set up just fine in azure portal and there are no obvious differeneces between the working client route and the not working signin route.

Cheers

Azure / application-gateway-kubernetes-ingress

502 error with all but one route in path-based routing #1527