search

Azure / kubernetes-keyvault-flexvol

Azure keyvault integration with Kubernetes via a Flex Volume
MIT License
253 stars 83 forks source link

Image pulling failed on one of the node in VMSS #139

Open mullerlele opened 4 years ago

mullerlele commented 4 years ago

Describe the bug installation failed on one of the nodes in the vmss, due to image pulling failures.

Steps To Reproduce 1) create a aks cluster with multiple vmss 2) provision more than 1 instance in one of the vmss (linux) 3) install flexvol based on the instructions 4) image pull back off (for the flexvol image) occurs on one of the node and hence the volume is not present on that node Expected behavior flexvol to be deployed as Daemonset with no problem

Key Vault FlexVolume version 0.0.15 Access mode: service principal or pod identity SP

Kubernetes version 1.14.7 Additional context

ritazh commented 4 years ago

Thanks for reporting this issue @mullerlele. I was not able to reproduce this issue on AKS with a 1.14.7 cluster. The image mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.15 was pulled successfully on each node. Can you pls try again?

mullerlele commented 4 years ago

Thank you @ritazh for your prompt reply. It still behaves the same - did you try it on a cluster with multiple vmss and aci as a virtual node? (the aci virtual node should not be of any concern in this case, because it is labeled not to have pod scheduled there, but I am only mentioning it here to provide a more detailed context)

ritazh commented 4 years ago

Hi @mullerlele I tried to repro this but could not. Could you pls provide more detailed repro steps. I followed the steps in this doc to create a new AKS cluster. Then ran the following cmds:

kubectl get no                                                                      
NAME                                STATUS   ROLES   AGE    VERSION
aks-nodepool1-36484348-vmss000000   Ready    agent   65m    v1.14.7
aks-nodepool1-36484348-vmss000001   Ready    agent   65m    v1.14.7
aks-nodepool1-36484348-vmss000002   Ready    agent   65m    v1.14.7
virtual-node-aci-linux              Ready    agent   6m8s   v1.13.1-vk-v0.9.0-1-g7b92d1ee-dev

kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml
namespace/kv created
daemonset.extensions/keyvault-flexvolume created

kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE                                NOMINATED NODE   READINESS GATES
kv            keyvault-flexvolume-7bsg7              1/1     Running   0          33s     10.240.0.69   aks-nodepool1-36484348-vmss000002   <none>           <none>
kv            keyvault-flexvolume-sv96c              1/1     Running   0          33s     10.240.0.40   aks-nodepool1-36484348-vmss000001   <none>           <none>
kv            keyvault-flexvolume-vl2j9              1/1     Running   0          33s     10.240.0.23   aks-nodepool1-36484348-vmss000000   <none>           <none>
...