search

Azure / application-gateway-kubernetes-ingress

This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
https://azure.github.io/application-gateway-kubernetes-ingress
MIT License
677 stars 420 forks source link

AGIC crashed at Applcation Gateway doesn't have a public IP #1400

Open galiacheng opened 2 years ago

galiacheng commented 2 years ago

Describe the bug AGIC 1.5.1 crashed when working with internal Application Gateway, error message:

F0531 10:05:48.200347       1 main.go:192] Got a fatal validation error on existing Application Gateway config. Please update Application Gateway or the controller's helm config. Error:Code="ErrorNoPublicIP" Message="Applcation Gateway doesn't have a public IP"

To Reproduce Steps to reproduce the behavior:

  1. Create internal Application Gateway: Follow Configure an application gateway with an internal load balancer (ILB) endpoint to provision Standard tier application gateway on subnet(10.3.0.0/28). The frontend IP is 10.3.0.6.

  2. Enable AGIC 1.5.1 See helm configuration:

    
# This file contains the essential configs for the ingress controller Helm chart

Verbosity level of the App Gateway Ingress Controller

verbosityLevel: 3

################################################################################

Specify which application gateway the ingress controller will manage

# appgw: subscriptionId: 260524c9-7a4d-4483-8d85-de54f9c40ae8 resourceGroup: haiche-aks-1-gateway-1 name: appgw1voa5jhws7maoc usePrivateIP: true

# Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
# This prohibits AGIC from applying config for any host/path.
# Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
shared: false

################################################################################

Specify which Kubernetes namespace the ingress controller will watch

Default value is "default"

Leaving this variable out or setting it to blank or empty string would

result in ingress controller observing all acessible namespaces.

# kubernetes: watchNamespace: sample-domain1-ns

################################################################################

Specify the authentication with Azure Resource Manager

#

Two authentication methods are available:

- Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)

armAuth:

type: aadPodIdentity

identityResourceID:

identityClientID:

armAuth: type: servicePrincipal secretJSON:

################################################################################

Specify if the cluster is RBAC enabled or not

rbac:

Specifies whether RBAC resources should be created

create: true


**Ingress Controller details**
* Output of `kubectl describe pod <ingress controller`> . The <ingress controller> pod name can be obtained by running `helm list`. 
```text
Name:         ingress-azure-7bb7749d8-q7bm7
Namespace:    default
Priority:     0
Node:         aks-agentpool-13946896-vmss000001/10.224.0.5
Start Time:   Tue, 31 May 2022 18:04:52 +0800
Labels:       app=ingress-azure
              pod-template-hash=7bb7749d8
              release=ingress-azure
Annotations:  checksum/config: 54f7501e40746d9d906a2ad5724979802a2e47f15ed8c3bad717d2c9cce9cf5c
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           10.244.1.12
IPs:
  IP:           10.244.1.12
Controlled By:  ReplicaSet/ingress-azure-7bb7749d8
Containers:
  ingress-azure:
    Container ID:   containerd://aaa26aa5dd34b3d4b465d99d09eee198b9df107628681f8760ee7e4865e54b74
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:cc131292df265926942e23ca5601a3de66e8feabcb81f705d8f7d84b740f81b6
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 31 May 2022 18:21:02 +0800
      Finished:     Tue, 31 May 2022 18:21:03 +0800
    Ready:          False
    Restart Count:  8
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  ingress-azure-7bb7749d8-q7bm7 (v1:metadata.name)
      AGIC_POD_NAMESPACE:             default (v1:metadata.namespace)
      AZURE_AUTH_LOCATION:            /etc/Azure/Networking-AppGW/auth/armAuth.json
    Mounts:
      /etc/Azure/Networking-AppGW/auth from networking-appgw-k8s-azure-service-principal-mount (ro)
      /etc/appgw/ from azure (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7nb6q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  Directory
  networking-appgw-k8s-azure-service-principal-mount:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  networking-appgw-k8s-azure-service-principal
    Optional:    false
  kube-api-access-7nb6q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  19m                   default-scheduler  Successfully assigned default/ingress-azure-7bb7749d8-q7bm7 to aks-agentpool-13946896-vmss000001
  Normal   Pulled     19m                   kubelet            Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 129.696054ms
  Normal   Pulled     19m                   kubelet            Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 100.958457ms
  Normal   Pulled     19m                   kubelet            Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 75.482394ms
  Normal   Created    19m (x4 over 19m)     kubelet            Created container ingress-azure
  Normal   Started    19m (x4 over 19m)     kubelet            Started container ingress-azure
  Normal   Pulling    19m (x4 over 19m)     kubelet            Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1"
  Normal   Pulled     19m                   kubelet            Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1" in 98.417938ms
  Warning  BackOff    4m54s (x82 over 19m)  kubelet            Back-off restarting failed container

goroutine 19 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x0) /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:1169 +0x6a created by k8s.io/klog/v2.init.0 /go/pkg/mod/k8s.io/klog/v2@v2.9.0/klog.go:420 +0xfb


* Any Azure support tickets associated with this issue.
peterkuiper commented 1 year ago

Any update on this? Issue #717 is related to this one. My solution does not allow for AppGW v2 so I'm stuck with v1. It seems moving the SKU validation would be my best bet although I'm not sure if this would be much work. @mscatyao Can you perhaps point me in the right direction?

kchervonets commented 1 year ago

@galiacheng I just got the same problem, did you find any solutions?

standaloneSA commented 11 months ago

Same problem here. Have a Standard_v2 application gateway with the EnableApplicationGatewayNetworkIsolation preview, and the controller is throwing

Error:Code="ErrorNoPublicIP" Message="Applcation Gateway doesn't have a public IP"

coolhome commented 10 months ago

I'm curious if the AGIC was never updated to handle a fully private appgw?

https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1423#issuecomment-1194455124

UPDATE - I discovered the AKS Addon was installing 1.5.3 while the private appgw was implemented in 1.7.0-RC. Upgrading kubernetes to 1.27.x brings AGIC Addon 1.7.x.

https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/CHANGELOG/CHANGELOG-1.7.md https://github.com/Azure/AKS/blob/master/CHANGELOG.md