Public IP HTTPS fails when creating private IP HTTPS ingress

[sprayzcs]

Describe the bug When creating a private ingress resource (annotation appgw.ingress.kubernetes.io/use-private-ip: "true"), all public https listeners fail with the error ERR_CONNECTION_RESET (from edge)

To Reproduce

• Create AGW with public and private ip
• Create public ingress with https
• Create private ingress with https

Result: The public https page returns the stated error while the private https page functions normally.
When turning off https on the public page, the public page loads normally.
When turning off https on the pivate page and turning on https on the public page, the public page works again with https.

I am using a custom CA to generate my TLS certificates, if that helps..

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
• Output of kubectl logs <ingress controller>:

  Name:             ingress-appgw-deployment-6b67ffdf9d-rq95z
  Namespace:        kube-system
  Priority:         0
  Service Account:  ingress-appgw-sa
  Node:             aks-system0000-37908770-vmss000007/10.1.0.5
  Start Time:       Mon, 13 Nov 2023 14:10:14 +0100
  Labels:           app=ingress-appgw
                kubernetes.azure.com/managedby=aks
                pod-template-hash=6b67ffdf9d
  Annotations:      checksum/config: 163d031e33ff447cec536dd3a56a52e88c0a227e1fe6fe40c0112a69b036c212
                cluster-autoscaler.kubernetes.io/safe-to-evict: true
                kubectl.kubernetes.io/restartedAt: 2023-11-13T14:10:13+01:00
                kubernetes.azure.com/metrics-scrape: true
                prometheus.io/path: /metrics
                prometheus.io/port: 8123
                prometheus.io/scrape: true
                resource-id:
                  /subscriptions/<xxx>/resourceGroups/<xxx>/providers/Microsoft.ContainerService/managedClusters/aks
  Status:           Running
  IP:               10.0.0.12
  IPs:
  IP:           10.0.0.12
  Controlled By:  ReplicaSet/ingress-appgw-deployment-6b67ffdf9d
  Containers:
  ingress-appgw-container:
  Container ID:   containerd://781c04692103eb02469401588073e0a2c92ba5f9a25ecd34ce8773945052708e
  Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
  Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
  Port:           <none>
  Host Port:      <none>
  State:          Running
    Started:      Mon, 13 Nov 2023 14:10:14 +0100
  Ready:          True
  Restart Count:  0
  Limits:
    cpu:     700m
    memory:  600Mi
  Requests:
    cpu:      100m
    memory:   20Mi
  Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
  Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
  Environment Variables from:
    ingress-appgw-cm  ConfigMap  Optional: false
  Environment:
    AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
    KUBERNETES_PORT_443_TCP_ADDR:   <xxx>.azmk8s.io
    KUBERNETES_PORT:                tcp://<xxx>.azmk8s.io:443
    KUBERNETES_PORT_443_TCP:        tcp://<xxx>.azmk8s.io:443
    KUBERNETES_SERVICE_HOST:        <xxx>.azmk8s.io
    AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
    AGIC_POD_NAME:                  ingress-appgw-deployment-6b67ffdf9d-rq95z (v1:metadata.name)
  Mounts:
    /etc/kubernetes/azure.json from cloud-provider-config (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jcb2h (ro)
  Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
  Volumes:
  cloud-provider-config:
  Type:          HostPath (bare host directory volume)
  Path:          /etc/kubernetes/azure.json
  HostPathType:  File
  kube-api-access-jcb2h:
  Type:                    Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds:  3607
  ConfigMapName:           kube-root-ca.crt
  ConfigMapOptional:       <nil>
  DownwardAPI:             true
  QoS Class:                   Burstable
  Node-Selectors:              <none>
  Tolerations:                 CriticalAddonsOnly op=Exists
                           node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                           node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                           node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:                      <none>

• Any Azure support tickets associated with this issue.

[akshaysngupta]

can you check what frontend all public ingresses are using on AppGateway ?

[sprayzcs]

can you check what frontend all public ingresses are using on AppGateway ?

There are two public and two private ingress resources. the first and third are my public ingresses, and the second and fourth are my private ingresses.

The public ingresses are using the public frontend ip.

Edit: I also noticed that shortly after the creation of the private ingress, both (public and private) ingresses function with https. After a short amount of time (~ 30 seconds), the public ingress does not work with https anymore.

[akshaysngupta]

@sprayzcs From the screenshot, is it right to conclude that AppGW config is generated as expected by AGIC but you see connectivity issue with AppGW ? If so, can you please create a support ticket ?