This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
Describe the bug
The token of Workload Identity cannot get acquired. Logs are reporting an incomplete environment variable configuration and in documentation no more possible values are specified.
Install AGIC using Helm (Version 1.7.1) with the following parameters:
appgw.applicationGatewayID: x
armAuth.type: workloadIdentity
armAuth.identityClientID: x
kubernetes.securityContext.runAsUser: 1000
rbac.enabled: true
verbosityLevel: 1
Ingress Controller details
Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
`Name: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6
Namespace: application-gateway-ingress-controller
Priority: 0
Service Account: application-gateway-kubernetes-ingress-sa-ingress-azure
Node: x
Start Time: Tue, 27 Jun 2023 07:43:46 +0200
Labels: app=ingress-azure
azure.workload.identity/use=true
pod-template-hash=54d9cf6f47
release=application-gateway-kubernetes-ingress
Annotations: checksum/config: fe8217e775b14e18c53cd5d54b2bc7719deda1da25a2f18345e9abaf101d1181
prometheus.io/port: 8123
prometheus.io/scrape: true
Status: Running
IP: x
IPs:
IP: x
Controlled By: ReplicaSet/application-gateway-kubernetes-ingress-ingress-azure-54d9cf6f47
Containers:
ingress-azure:
Container ID: containerd://6bde7e11a42e5c995fabd9f55ba9843ce7a2c51fdae45d5fa5e36ef8eacdeff2
Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1
Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:91a6648b78c65f3b6858441589daabd72146d9a53e896c0e6abf501e870f9d9b
Port:
Host Port:
State: Running
Started: Tue, 27 Jun 2023 07:43:47 +0200
Ready: False
Restart Count: 0
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
application-gateway-kubernetes-ingress-cm-ingress-azure ConfigMap Optional: false
Environment:
AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
AGIC_POD_NAME: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 (v1:metadata.name)
AGIC_POD_NAMESPACE: application-gateway-ingress-controller (v1:metadata.namespace)
AZURE_CLIENT_ID: x
AZURE_TENANT_ID: x
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
Mounts:
/etc/appgw/ from azure (ro)
/var/run/secrets/azure/tokens from azure-identity-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fbjhm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
azure:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/
HostPathType: Directory
kube-api-access-fbjhm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
azure-identity-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
QoS Class: BestEffort
Node-Selectors: x
Tolerations: :NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal Scheduled 56m default-scheduler Successfully assigned application-gateway-ingress-controller/application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 to x
Normal Pulling 56m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1"
Normal Pulled 56m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1" in 261.149933ms (261.155633ms including waiting)
Normal Created 56m kubelet Created container ingress-azure
Normal Started 56m kubelet Started container ingress-azure
Warning Unhealthy 67s (x374 over 55m) kubelet Readiness probe failed: Get "http://x:8123/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
`
Output of kubectl logs <ingress controller>. I0627 05:43:47.609870 1 utils.go:114] Using verbosity level 1 from environment variable APPGW_VERBOSITY_LEVEL
I0627 05:43:47.616327 1 main.go:81] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file "/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied
I0627 05:43:48.648152 1 supported_apiversion.go:70] server version is: 1.25.6
I0627 05:43:48.659580 1 environment.go:294] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0627 05:43:48.659607 1 main.go:118] Using User Agent Suffix='application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6' when communicating with ARM
I0627 05:43:48.659688 1 auth.go:58] Creating authorizer using Default Azure Credentials
I0627 05:43:48.660007 1 httpserver.go:57] Starting API Server on :8123
E0627 05:44:48.660931 1 authorizer.go:46] Error getting Azure token: DefaultAzureCredential: failed to acquire a token.
Attempted credentials:
EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set
WorkloadIdentityCredential: unable to resolve an endpoint: server response error:
context deadline exceeded
`
Describe the bug The token of Workload Identity cannot get acquired. Logs are reporting an incomplete environment variable configuration and in documentation no more possible values are specified.
The configuration is identical to the Testing section in the PR https://github.com/Azure/application-gateway-kubernetes-ingress/pull/1498 of @akshaysngupta
To Reproduce Steps to reproduce the behavior:
Install AGIC using Helm (Version 1.7.1) with the following parameters:
appgw.applicationGatewayID: x armAuth.type: workloadIdentity armAuth.identityClientID: x kubernetes.securityContext.runAsUser: 1000 rbac.enabled: true verbosityLevel: 1
Ingress Controller details
Output of pod name can be obtained by running
Host Port:
State: Running
Started: Tue, 27 Jun 2023 07:43:47 +0200
Ready: False
Restart Count: 0
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
application-gateway-kubernetes-ingress-cm-ingress-azure ConfigMap Optional: false
Environment:
AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
AGIC_POD_NAME: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 (v1:metadata.name)
AGIC_POD_NAMESPACE: application-gateway-ingress-controller (v1:metadata.namespace)
AZURE_CLIENT_ID: x
AZURE_TENANT_ID: x
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
Mounts:
/etc/appgw/ from azure (ro)
/var/run/secrets/azure/tokens from azure-identity-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fbjhm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
azure:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/
HostPathType: Directory
kube-api-access-fbjhm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
azure-identity-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
QoS Class: BestEffort
Node-Selectors: x
Tolerations: :NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
kubectl describe pod <ingress controller> . Thehelm list. `Name: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 Namespace: application-gateway-ingress-controller Priority: 0 Service Account: application-gateway-kubernetes-ingress-sa-ingress-azure Node: x Start Time: Tue, 27 Jun 2023 07:43:46 +0200 Labels: app=ingress-azure azure.workload.identity/use=true pod-template-hash=54d9cf6f47 release=application-gateway-kubernetes-ingress Annotations: checksum/config: fe8217e775b14e18c53cd5d54b2bc7719deda1da25a2f18345e9abaf101d1181 prometheus.io/port: 8123 prometheus.io/scrape: true Status: Running IP: x IPs: IP: x Controlled By: ReplicaSet/application-gateway-kubernetes-ingress-ingress-azure-54d9cf6f47 Containers: ingress-azure: Container ID: containerd://6bde7e11a42e5c995fabd9f55ba9843ce7a2c51fdae45d5fa5e36ef8eacdeff2 Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1 Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:91a6648b78c65f3b6858441589daabd72146d9a53e896c0e6abf501e870f9d9b Port:Normal Scheduled 56m default-scheduler Successfully assigned application-gateway-ingress-controller/application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 to x Normal Pulling 56m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1" Normal Pulled 56m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1" in 261.149933ms (261.155633ms including waiting) Normal Created 56m kubelet Created container ingress-azure Normal Started 56m kubelet Started container ingress-azure Warning Unhealthy 67s (x374 over 55m) kubelet Readiness probe failed: Get "http://x:8123/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) `
Output of
kubectl logs <ingress controller>.I0627 05:43:47.609870 1 utils.go:114] Using verbosity level 1 from environment variable APPGW_VERBOSITY_LEVEL I0627 05:43:47.616327 1 main.go:81] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file "/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied I0627 05:43:48.648152 1 supported_apiversion.go:70] server version is: 1.25.6 I0627 05:43:48.659580 1 environment.go:294] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces. I0627 05:43:48.659607 1 main.go:118] Using User Agent Suffix='application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6' when communicating with ARM I0627 05:43:48.659688 1 auth.go:58] Creating authorizer using Default Azure Credentials I0627 05:43:48.660007 1 httpserver.go:57] Starting API Server on :8123 E0627 05:44:48.660931 1 authorizer.go:46] Error getting Azure token: DefaultAzureCredential: failed to acquire a token. Attempted credentials: EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set WorkloadIdentityCredential: unable to resolve an endpoint: server response error: context deadline exceeded `Azure Support Request ID: 2306270050000735