JoeyC-Dev
commented
4 months ago Based on your error message "Failed to get IP address from CNS with error", it looks like you are using "Azure CNI for dynamic IP allocation". So I did a set-up like below:
# Basic parameter
ranNum=$(echo $RANDOM)
rG=aks-test-${ranNum}
aks=aks-test-${ranNum}
aksVer=1.29.4
logAnalyticsWorkspace=aks-law-${ranNum}
location=southeastasia
# AKS set-up (Azure CNI for dynamic IP allocation)
az group create -n ${rG} -l ${location} -o none
groupID=$(az group show -n ${rG} --query id -o tsv)
az monitor log-analytics workspace create -n ${logAnalyticsWorkspace} -g ${rG} -o none
logAnalyticsWorkspaceId=$(az resource show -n ${logAnalyticsWorkspace} -g ${rG} \
--namespace Microsoft.OperationalInsights --resource-type workspaces --query id -o tsv)
az network vnet create -g ${rG} -n aks-vnet --address-prefixes 10.0.0.0/8 -o none
az network vnet subnet create -g ${rG} --vnet-name aks-vnet --name node1 --address-prefixes 10.240.0.0/16 -o none
az network vnet subnet create -g ${rG} --vnet-name aks-vnet --name pod1 --address-prefixes 10.241.0.0/16 -o none
az aks create -n ${aks} -g ${rG} --node-vm-size Standard_A4_v2 --node-count 2 \
-a monitoring --workspace-resource-id ${logAnalyticsWorkspaceId} --enable-syslog \
--vnet-subnet-id ${groupID}/providers/Microsoft.Network/virtualNetworks/aks-vnet/subnets/node1 \
--pod-subnet-id ${groupID}/providers/Microsoft.Network/virtualNetworks/aks-vnet/subnets/pod1 \
--network-plugin azure --kubernetes-version ${aksVer} --node-os-upgrade-channel None --no-ssh-key -o none
# Deploy example deployment
az aks get-credentials -n ${aks} -g ${rG}
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
EOF
# Debug info
CNSinfo=$(kubectl get ds azure-cns -n kube-system -o json | jq -r '.spec.template.spec.containers' | jq '[.[]|select(.name=="cns-container")][0].image')
echo "Your CNS image info: ${CNSinfo}"
# My result: "mcr.microsoft.com/containernetworking/azure-cns:v1.5.26"I completed setting up my AKS around 2024-06-08T17:43:24Z (when nginx-deployment has been deployed), and then find no error at all.
No abnormal events on Pods:
After then, I just go set another script to repeatedly delete and create Pods:
#!/bin/bash
i=1
while [ $i -ge 1 ]
do
echo "Nginx deployment is being created ${i} time(s)."
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
EOF
sleep 40; kubectl delete deployment nginx-deployment; ((i+=1)); sleep 20
doneThen I go to sleep and back to see what happened after then.
Nothing wrong: (Remember the time I mentioned before? These errors are the ones appears during the initial set-up time, but never popped again later)
So far, can see, it looks like something related to your network architecture. At least from initial set-up sight, there is nothing wrong by now.
It is suggested you provide more information about:
- How you set up your AKS (incl. network setting like UDR and sth else)
- How many Pods running averagely every day
- Issue pattern (time, frequency)
- Your CNS version
- More possible logs that collected from CNS, syslog, azure-vnet, kube-audit, etc.
The info you provided is not enough to get where the issue is, as you see in this reply. If this is not reproducible issue, you probably have to open a support ticket.
But since you have this issue among all 4 AKS instances, so it is possible that this is reproducible, but the info is not enough to reproducible it.
If you have raised a support request, you can also mention it here. Because there is still a chance that this issue is not-reproducible outside your environment. It is suggested to open a support ticket to let support engineer see what is happening.
kevinharing
mortenjoenby
microsoft-github-policy-service[bot]
Describe the bug Since around 2024-05-08T12:00:00Z we're seeing the below error popping up in the kube events table a lot. This error pops up on all our clusters (4 in total). It happens on all namespaces and it seems related to spinning up the pods and the node not being able to connect to a certain service running on localhost.
To Reproduce Run a deployment.
Expected behavior No errors should occur when running a deployment.
Screenshots If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):