search

Azure / application-gateway-kubernetes-ingress

This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
https://azure.github.io/application-gateway-kubernetes-ingress
MIT License
677 stars 420 forks source link

AGIC is not detecting ingressClassName when is set in a Ingress manifest #1565

Open carlosbustillordguez opened 1 year ago

carlosbustillordguez commented 1 year ago

Describe the bug I have installed AGIC v1.7.2 with the following values through Helm:

USER-SUPPLIED VALUES:
appgw:
  name: internalGateway-aksCluster
  resourceGroup: MC_aksTerra_aksCluster_westeurope
  subnetID: /subscriptions/****/resourceGroups/vnet-testing/providers/Microsoft.Network/virtualNetworks/testVNet/subnets/appGatewaySubnet
  subscriptionId: ****
  usePrivateIP: "true"
armAuth:
  identityClientID: 164b09fd-f79a-4708-a26f-4940870cc811
  type: workloadIdentity
image:
  tag: 1.7.2
kubernetes:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: kubernetes.azure.com/mode
            operator: In
            values:
            - system
        weight: 100
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.azure.com/cluster
            operator: Exists
          - key: type
            operator: NotIn
            values:
            - virtual-kubelet
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
  ingressClass: azure-application-gateway-internal
  ingressClassResource:
    controllerValue: azure/application-gateway
    default: false
    enabled: true
    name: azure-application-gateway-internal
  nodeSelector: {}
  tolerations: []
rbac:
  enabled: true
verbosityLevel: 3

When is applied the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
spec:
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9797"
      labels:
        app: podinfo
    spec:
      containers:
      - name: podinfod
        # image: ghcr.io/stefanprodan/podinfo:6.0.3
        image: stefanprodan/podinfo:6.1.6
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9898
          protocol: TCP
        - name: http-metrics
          containerPort: 9797
          protocol: TCP
        - name: grpc
          containerPort: 9999
          protocol: TCP
        command:
        - ./podinfo
        - --port=9898
        - --port-metrics=9797
        - --grpc-port=9999
        - --grpc-service-name=podinfo
        - --level=info
        - --random-delay=false
        - --random-error=false
        env:
        - name: PODINFO_UI_COLOR
          value: "#34577c"
        livenessProbe:
          exec:
            command:
            - podcli
            - check
            - http
            - localhost:9898/healthz
          initialDelaySeconds: 5
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - podcli
            - check
            - http
            - localhost:9898/readyz
          initialDelaySeconds: 5
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 200m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 9898
  - name: grpc
    port: 9999
    protocol: TCP
    targetPort: grpc
  selector:
    app: podinfo
---
# Private Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: podinfo-private
  annotations:
    appgw.ingress.kubernetes.io/use-private-ip: "true"
    # kubernetes.io/ingress.class: azure-application-gateway-internal
spec:
  ingressClassName: azure-application-gateway-internal
  rules:
    - host: podinfo-private.mydomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: podinfo
                port:
                  number: 80

The AGIC doesn't detect the Ingress (note the "ADDRESS" column is empty):

k get ingress
NAME              CLASS                                HOSTS                          ADDRESS   PORTS   AGE
podinfo-private   azure-application-gateway-internal   podinfo-private.mydomain.com             80      22m

To Reproduce Steps to reproduce the behavior:

  1. Install the AGIC through Helm using the above user-supplied values.
  2. Apply the above manifest.
  3. Get Ingress objects, and note that the "ADDRESS" column is empty for the podinfo-private Ingress.
  4. Then, edit the podinfo-private Ingress and add the following annotation: 
       kubernetes.io/ingress.class: azure-application-gateway-internal
  5. Get the ingress objects, and now note that the "ADDRESS" column has the private IP address of the Ingress (LB). For instance: 
       k get ingress
   NAME              CLASS                                HOSTS                          ADDRESS        PORTS   AGE
   podinfo-private   azure-application-gateway-internal   podinfo-private.mydomain.com   10.12.17.254   80      27m

    Ingress Controller details

    • Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list. 
      k -n kube-system describe pod ingress-azure-internal-67c55ff6c6-5rg4q
Name:             ingress-azure-internal-67c55ff6c6-5rg4q
Namespace:        kube-system
Priority:         0
Service Account:  ingress-azure-internal
Node:             aks-agentpool-57653592-vmss000000/10.12.0.4
Start Time:       Sat, 22 Jul 2023 14:04:57 +0200
Labels:           app=ingress-azure
          azure.workload.identity/use=true
          pod-template-hash=67c55ff6c6
          release=ingress-azure-internal
Annotations:      checksum/config: 41df885cad3267338ef942c5de29b0943e1528d38ac3bc1f36d371591102e229
          prometheus.io/port: 8123
          prometheus.io/scrape: true
Status:           Running
IP:               10.12.0.74
IPs:
IP:           10.12.0.74
Controlled By:  ReplicaSet/ingress-azure-internal-67c55ff6c6
Containers:
ingress-azure:
Container ID:   containerd://5f77d7d52abb83195917fc9a75a071266a8a6b612bbfcc6c0972d5c694a8a02c
Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
Port:           <none>
Host Port:      <none>
State:          Running
Started:      Sat, 22 Jul 2023 14:05:02 +0200
Ready:          True
Restart Count:  0
Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
ingress-azure-internal  ConfigMap  Optional: false
Environment:
AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
AGIC_POD_NAME:                  ingress-azure-internal-67c55ff6c6-5rg4q (v1:metadata.name)
AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
KUBERNETES_PORT_443_TCP_ADDR:   aksterra-t53lt2qz.hcp.westeurope.azmk8s.io
KUBERNETES_PORT:                tcp://aksterra-t53lt2qz.hcp.westeurope.azmk8s.io:443
KUBERNETES_PORT_443_TCP:        tcp://aksterra-t53lt2qz.hcp.westeurope.azmk8s.io:443
KUBERNETES_SERVICE_HOST:        aksterra-t53lt2qz.hcp.westeurope.azmk8s.io
AZURE_CLIENT_ID:                164b09fd-f79a-4708-a26f-4940870cc811
AZURE_TENANT_ID:                e878867c-44ef-4567-b4d5-c261f396c4e0
AZURE_FEDERATED_TOKEN_FILE:     /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST:           https://login.microsoftonline.com/
Mounts:
/etc/appgw/ from azure (ro)
/var/run/secrets/azure/tokens from azure-identity-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j25m7 (ro)
Conditions:
Type              Status
Initialized       True
Ready             True
ContainersReady   True
PodScheduled      True
Volumes:
azure:
Type:          HostPath (bare host directory volume)
Path:          /etc/kubernetes/
HostPathType:  Directory
kube-api-access-j25m7:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  3607
ConfigMapName:           kube-root-ca.crt
ConfigMapOptional:       <nil>
DownwardAPI:             true
azure-identity-token:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  3600
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                     node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type    Reason     Age   From               Message
----    ------     ----  ----               -------
Normal  Scheduled  37m   default-scheduler  Successfully assigned kube-system/ingress-azure-internal-67c55ff6c6-5rg4q to aks-agentpool-57653592-vmss000000
Normal  Pulling    37m   kubelet            Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2"
Normal  Pulled     37m   kubelet            Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2" in 3.731921935s (3.731931035s including waiting)
Normal  Created    37m   kubelet            Created container ingress-azure
Normal  Started    37m   kubelet            Started container ingress-azure
    • Output of kubectl logs <ingress controller>. **Controller logs when is set onlyingressClassName `:** 
      I0722 12:05:02.303079       1 utils.go:114] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
I0722 12:05:02.341546       1 supported_apiversion.go:70] server version is: 1.26.3
I0722 12:05:02.353112       1 environment.go:294] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0722 12:05:02.353131       1 main.go:118] Using User Agent Suffix='ingress-azure-internal-67c55ff6c6-5rg4q' when communicating with ARM
I0722 12:05:02.353249       1 main.go:137] Application Gateway Details: Subscription="*********" Resource Group="MC_aksTerra_aksCluster_westeurope" Name="internalGateway-aksCluster"
I0722 12:05:02.353319       1 httpserver.go:57] Starting API Server on :8123
I0722 12:05:02.353633       1 auth.go:58] Creating authorizer using Default Azure Credentials
I0722 12:05:02.762592       1 main.go:184] Ingress Controller will observe all namespaces.
I0722 12:05:02.824897       1 context.go:171] k8s context run started
I0722 12:05:02.824930       1 context.go:238] Waiting for initial cache sync
I0722 12:05:02.825008       1 reflector.go:219] Starting reflector *v1.Secret (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825018       1 reflector.go:255] Listing and watching *v1.Secret from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825197       1 reflector.go:219] Starting reflector *v1.Endpoints (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825208       1 reflector.go:255] Listing and watching *v1.Endpoints from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825317       1 reflector.go:219] Starting reflector *v1.Pod (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825344       1 reflector.go:255] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825445       1 reflector.go:219] Starting reflector *v1.Service (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825454       1 reflector.go:255] Listing and watching *v1.Service from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825560       1 reflector.go:219] Starting reflector *v1beta1.AzureApplicationGatewayRewrite (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825569       1 reflector.go:255] Listing and watching *v1beta1.AzureApplicationGatewayRewrite from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825751       1 reflector.go:219] Starting reflector *v1.Ingress (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825761       1 reflector.go:255] Listing and watching *v1.Ingress from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825937       1 reflector.go:219] Starting reflector *v1.IngressClass (30s) from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.825947       1 reflector.go:255] Listing and watching *v1.IngressClass from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167
I0722 12:05:02.925466       1 context.go:251] Initial cache sync done
I0722 12:05:02.925482       1 context.go:252] k8s context run finished
I0722 12:05:02.925533       1 worker.go:39] Worker started
I0722 12:05:03.268780       1 mutate_app_gateway.go:166] BEGIN AppGateway deployment
I0722 12:05:04.013508       1 client.go:220] OperationID='1efc0421-7329-4ad8-91d2-34f8fca93a45'
I0722 12:05:09.063582       1 mutate_app_gateway.go:174] Applied generated Application Gateway configuration
I0722 12:05:09.063648       1 mutate_app_gateway.go:189] cache: Updated with latest applied config.
I0722 12:05:09.064227       1 mutate_app_gateway.go:193] END AppGateway deployment
I0722 12:05:09.064269       1 controller.go:152] Completed last event loop run in: 6.138724096s
I0722 12:11:32.387527       1 mutate_app_gateway.go:153] cache: Config has NOT changed! No need to connect to ARM.
I0722 12:11:32.387548       1 controller.go:152] Completed last event loop run in: 104.469012ms

Controller logs when the annotation is set:

I0722 12:38:38.797922       1 backendhttpsettings.go:89] Created backend http settings bp-default-podinfo-80-9898-podinfo-private for ingress default/podinfo-private and service default/podinfo
I0722 12:38:38.806754       1 mutate_app_gateway.go:166] BEGIN AppGateway deployment
I0722 12:38:40.221275       1 client.go:220] OperationID='2d17a5c9-82c4-4c00-83aa-fbcd4e9d6af3'
I0722 12:38:45.272202       1 mutate_app_gateway.go:174] Applied generated Application Gateway configuration
I0722 12:38:45.272224       1 mutate_app_gateway.go:189] cache: Updated with latest applied config.
I0722 12:38:45.272953       1 mutate_app_gateway.go:193] END AppGateway deployment
I0722 12:38:45.272967       1 controller.go:152] Completed last event loop run in: 6.573533722s
I0722 12:38:46.313598       1 context.go:748] IP 10.12.17.254 already set on Ingress default/podinfo-private
I0722 12:38:46.314741       1 backendhttpsettings.go:89] Created backend http settings bp-default-podinfo-80-9898-podinfo-private for ingress default/podinfo-private and service default/podinfo
I0722 12:38:46.320227       1 mutate_app_gateway.go:153] cache: Config has NOT changed! No need to connect to ARM.
I0722 12:38:46.320246       1 controller.go:152] Completed last event loop run in: 47.140569ms
carlosbustillordguez commented 1 year ago

Any update about this??

akshaysngupta commented 1 year ago 
  ingressClass: azure-application-gateway-internal
  ingressClassResource:
    controllerValue: azure/application-gateway
    default: false
    enabled: true
    name: azure-application-gateway-internal

can you remove ingressClass: azure-application-gateway-internal from above spec and try ?

carlosbustillordguez commented 1 year ago

Hi @akshaysngupta! Thank you for your response. I followed your recommendation and now is working:

k get ingress
NAME               CLASS                                HOSTS                           ADDRESS        PORTS   AGE
podinfo-private    azure-application-gateway-internal   podinfo-private.mydomain.com    10.12.17.254   80      19m

From your suggestion, I can understand that if ingressClass: azure-application-gateway-internal is present has a higher priority over ingressClassResource.name, right? What about if I want to use my custom class azure-application-gateway-internal in the annotation and by defining spec.ingressClassName in the Ingress definition?