K8 App GW Ingress Controller is removing SSL Certificate in Azure

erikschlegel commented 4 years ago

Describe the bug App Gateway Keyvault-backed SSL certificate is removed after ingress controller pod is bootstrapped on the cluster

I'm trying to use the appgw.ingress.kubernetes.io/appgw-ssl-certificate: ssl_cert annotation to reference the KV backed cert.

To Reproduce

Associate a keyvault certificate to an application gateway instance
deploy the app-gw-controller helm chart
Validate the certificate reference in app-gw is now removed. az network application-gateway ssl-cert list --resource-group erischdev-6g-6g0zcuvy-aks-rg --gateway-name erischdev-6g0zcuvy-osdu-r2-appgw

Ingress Controller details

kubectl describe pod -n app-gw appgw-controller-ingress-azure-656696c799-npt6x
Name:           appgw-controller-ingress-azure-656696c799-npt6x
Namespace:      app-gw
Priority:       0
Node:           aks-default-23646475-vmss000001/10.10.1.35
Start Time:     Tue, 14 Apr 2020 14:19:13 -0500
Labels:         aadpodidbinding=appgw-controller-ingress-azure
                app=ingress-azure
                pod-template-hash=656696c799
                release=appgw-controller
Annotations:    prometheus.io/port: 8123
                prometheus.io/scrape: true
Status:         Running
IP:             10.10.1.54
IPs:            <none>
Controlled By:  ReplicaSet/appgw-controller-ingress-azure-656696c799
Containers:
  ingress-azure:
    Container ID:   docker://928881ca8fb535069787d8321b48b170ef7a76a84d53a8665254e299d7e03a29
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c295f99ae66443c5a392fd894620fcd1fc313b9efdec96d13f166fefb29780a9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 14 Apr 2020 14:19:14 -0500
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      appgw-controller-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  appgw-controller-ingress-azure-656696c799-npt6x (v1:metadata.name)
      AGIC_POD_NAMESPACE:             app-gw (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from appgw-controller-sa-ingress-azure-token-fmlsn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  appgw-controller-sa-ingress-azure-token-fmlsn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  appgw-controller-sa-ingress-azure-token-fmlsn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                                      Message
  ----    ------     ----   ----                                      -------
  Normal  Scheduled  9m57s  default-scheduler                         Successfully assigned app-gw/appgw-controller-ingress-azure-656696c799-npt6x to aks-default-23646475-vmss000001
  Normal  Pulling    9m57s  kubelet, aks-default-23646475-vmss000001  Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
  Normal  Pulled     9m56s  kubelet, aks-default-23646475-vmss000001  Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
  Normal  Created    9m56s  kubelet, aks-default-23646475-vmss000001  Created container ingress-azure
  Normal  Started    9m56s  kubelet, aks-default-23646475-vmss000001  Started container ingress-azure

Output of `kubectl logs .

kubectl logs appgw-controller-ingress-azure-656696c799-npt6x -n app-gw
ERROR: logging before flag.Parse: I0414 19:19:14.363863       1 main.go:302] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
I0414 19:19:14.411006       1 main.go:102] Unable to load Azure Context file:
I0414 19:19:14.411023       1 main.go:132] App Gateway Details: Subscription: 929e9ae0-7bb1-4563-a200-9863fe27cae4, Resource Group: erischdev-6g-6g0zcuvy-aks-rg, Name: erischdev-6g0zcuvy-osdu-r2-appgw
I0414 19:19:14.411038       1 auth.go:90] Creating authorizer from Azure Managed Service Identity
I0414 19:19:15.395597       1 main.go:181] Ingress Controller will observe the following namespaces:app-gw
I0414 19:19:15.430425       1 context.go:129] k8s context run started
I0414 19:19:15.430474       1 context.go:168] Waiting for initial cache sync
I0414 19:19:15.530914       1 context.go:176] Initial cache sync done
I0414 19:19:15.531064       1 context.go:177] k8s context run finished
I0414 19:19:15.531413       1 httpserver.go:57] Starting API Server on :8123
I0414 19:19:15.531418       1 worker.go:35] Worker started
I0414 19:19:15.698198       1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I0414 19:19:36.266410       1 mutate_app_gateway.go:182] Applied App Gateway config in 20.568183214s
I0414 19:19:36.266446       1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I0414 19:19:36.267350       1 mutate_app_gateway.go:203] END AppGateway deployment

3quanfeng commented 4 years ago

Hi @erikschlegel, May I ask which version of agic are you using?

erikschlegel commented 4 years ago

Hi @erikschlegel, May I ask which version of agic are you using?

I'm using the latest helm chart in master at https://github.com/Azure/application-gateway-kubernetes-ingress/helm/ingress-azure

For App Gateway, I'm using the WAF_V2 sku

For additional context, I'm using the Ingress Controller as suggested in the readme

3quanfeng commented 4 years ago

Hi @erikschlegel, thanks! Could you please help bootstrap agic first and then configure the key-vault ssl-certificate to appgw to use the new annotation? I am guessing if the issue is caused by the reconciler.

erikschlegel commented 4 years ago

Hi @erikschlegel, thanks! Could you please help bootstrap agic first and then configure the key-vault ssl-certificate to appgw to use the new annotation? I am guessing if the issue is caused by the reconciler.

I ran into issues as my Terraform deployment(which generates the KV SSL cert) overwrites the K8-specific app-gw routes.

3quanfeng commented 4 years ago

Hi @erikschlegel , btw, make sure you use "--version 1.2.0-rc1" when installing with the helm, helm install by default installs only the latest release version, e.g. helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --version 1.2.0-rc1 --generate-name

erikschlegel commented 4 years ago

Thanks for the feedback. Confirmed this issue is resolved when pointing my helm version to 1.2.0-rc1 thanks for the help @3quanfeng

sudhakar890 commented 4 years ago

I'm using AGIC version 1.2.0-rc3 and application gateway ssl certificates are getting removed by AGIC.

akshaysngupta commented 4 years ago

@sudhakar It could be that you have two AGIC deployments in your cluster targeting the same Application Gateway. Can you check if that's that case ? You can use helm list to see your current deployments.

Azure / application-gateway-kubernetes-ingress

K8 App GW Ingress Controller is removing SSL Certificate in Azure #807