search

Azure / ACS

Azure Container Service - Bug Tracker + Announcements
65 stars 27 forks source link

AttachDetach Failing for k8s 1.7.7 #118

Closed smileisak closed 5 years ago

smileisak commented 6 years ago

Is this a request for help?: It is a request.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Is this a BUG REPORT

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

What happened:

I'm in production environment, running some databases as a StatefulSets with dynamic provisioning. Since yesterday, some pods fails with CrachLoopBackOff state.

This is the events from my Kafka pod:

Events:
  Type     Reason       Age                  From                           Message
  ----     ------       ----                 ----                           -------
  Warning  FailedMount  4m (x260 over 20h)   attachdetach                   AttachVolume.Attach failed for volume "pvc-efec7085-1250-******" : Attach volume "*****-****-4365d8mg-pvc-efec7085-1250-11e8-***-****" to instance "k8s-agent-****" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with status 'Failed': Code="AcquireDiskLeaseFailed" Message="Failed to acquire lease while creating disk 'smartcoach-k8s-smartcoach-4365d8mg-pvc-efec7085-1250-11e8-976e-000d3a2ed6a2' using blob with URI https://****.blob.core.windows.net/1634572225/****-k8s-****-****-pvc-****-1250-11e8-976e-*****.vhd. Blob is already in use."
  Warning  FailedMount  59s (x545 over 20h)  kubelet, k8s-agent-b91c42af-4  Unable to mount volumes for pod "kafka-*****-***(05471b68-6d67-11e8-a318-000d3a2ed670)": timeout expired waiting for volumes to attach/mount for pod "****-***"/"kafka-0". list of unattached/unmounted volumes=[data]
  Warning  FailedSync   59s (x545 over 20h)  kubelet, k8s-agent-****-4  Error syncing pod

What you expected to happen:

When pods fails, it will reschedule in another node, and volume should detach and attach successfully.

How to reproduce it (as minimally and precisely as possible):

It was produced suddenly, i can't tell exactlly how to reproduce it. I think it is related to Multiple Azure Services impacted in West Europe : https://azure.microsoft.com/en-us/status/history/

Anything else we need to know: It is working fine before this bug. I mean detaching and attaching volumes was working perfectly.

andyzhangx commented 6 years ago

Your issue could be related to https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md#1-disk-attach-error

Could you help check whether disk https://****.blob.core.windows.net/1634572225/****-k8s-****-****-pvc-****-1250-11e8-976e-*****.vhd is already attached to another VM? You may also try the mitigation way in the above link.

andyzhangx commented 6 years ago

@smileisak is this issue resolved?

jbouzekri commented 6 years ago

@andyzhangx : I work with @smileisak. Yes it has been resolved.

After last sunday Azure crash in West Europe datacenter, when the cluster started up again, a few disks were attached to the wrong VM (sometimes to 2 VMs at the same time). After detaching those disks from all the servers they were on, the pod syncs again on its own and attach the disk to the correct server (note it can take a few minutes sometimes where the pod says something like "sync error timeout")

Thank you for your link and the question about whether the disk was attached to another VM. It was very helpful to us.

Note : I created a small script to detach disks that are mounted on multiple VMs https://gist.github.com/jbouzekri/11c56ed5ebb038e6166300ed294630e7