Busy azure-disk regularly fail to mount causing K8S Pod deployments to halt.

dbalaouras commented 7 years ago

I've setup Azure Container Service with Kubernetes and I use dynamic provisioning of volumes (see details below) when deploying new Pods. Quite frequently (about 10%) I get the following error which halts the deployment:

14h 1m 439 {controller-manager } Warning FailedMount Failed to attach volume "pvc-95aa8dbf-082e-11e7-af1a-000d3a2735d9" on node "k8s-agent-1da8a8df-2" with: Attach volume "clst-west-eu-dev-dynamic-pvc-95aa8dbf-082e-11e7-af1a-000d3a2735d9.vhd" to instance "k8s-agent-1DA8A8DF-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'clst-west-eu-dev-dynamic-pvc-f843f8fa-0663-11e7-af1a-000d3a2735d9.vhd' to VM 'k8s-agent-1DA8A8DF-2' because the disk is currently being detached. Please wait until the disk is completely detached and then try again."

The Pod deployment then halts forever, or until I delete the Pod and let the ReplicationController create a new one.

Any idea what is causing this?

Workflow

I have created the following StorageClass:

Name:       azure-disk
IsDefaultClass: No
Annotations:    <none>
Provisioner:    kubernetes.io/azure-disk
Parameters: location=westeu,skuName=Standard_LRS,storageAccount=<<storageaccount>>

The storageaccount does contain a Blob service named vhds.

When deploying a new Pod, I create a PVC that looks like this:

{
  "apiVersion": "v1",
  "kind": "PersistentVolumeClaim",
  "Provisioner": "kubernetes.io/azure-disk",
  "metadata": {
    "name": "test-deployment-pvc",
    "annotations": {
      "volume.beta.kubernetes.io/storage-class": "azure-disk"
    },
    "labels": {
      "org": "somelabel"
    }
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "1Gi"
      }
    }
  }
}

and finally use the PVC in the pods:

{
  "volumes": [
    {
      "persistentVolumeClaim": {
        "claimName": "test-deployment-pvc"
      },
      "name": "storage"
    }
  ]
}

JackQuincy commented 7 years ago

Sorry for the slow response time. There is a known performance issue here with the azure linux drivers. the kubernetes repo might have more expertise on this. But I do know that Kubernetes 1.6 has a speed up for attaching the disks. A work around we have asked people to use in the short term is to preformat the disks to ext4. Now how to do that in the PVC I'm not the most familiar with. @colemickens did I get this right?

colemickens commented 7 years ago

It's not the ext4 format being slow (which is addressed in 1.5.3+) since the PV was provisioned and in use, but rather wasn't able to detached and reattached when rescheduled.

@dbalaouras, I assume this is continuing to happen, even after 4-5 minutes? If not, as @JackQuincy indicated, the attach code has seen improvements in recent releases (1.5.3+, 1.6), and a full rewrite is coming in the future: https://github.com/kubernetes/kubernetes/pull/41950

This should probably be filed as an upstream Issue on Kubernetes and diagnosed further there. @dbalaouras can you do that and link it here? It would be helpful if you can repro and include these things:

The instanceView of the VM where the Pod was originally scheduled (where the detach is occurring) (hostA)
The instanceView of the VM where the Pod has been rescheduled (where the attach is occurring) (hostB)
Confirm that just deleting the Pod is sufficient to repair. In this case, does it work because the Pod lands back on (hostA) when rescheduled?

The instanceViews can be collected from https://resources.azure.com. Navigate through the hierarchy to the VM and then go to the child node "instanceView". You can also get it from the CLI: az vm get-instance-view --resource-group $RG --name $VM

dbalaouras commented 7 years ago

Hi @colemickens and @JackQuincy - thanks for the feedback.

Indeed, this error happens every 4-5 minutes; i.e. I see the following log:

Failed to attach volume "pvc-c209421d-12dd-11e7-af1a-000d3a2735d9" on node "k8s-agent-1da8a8df-2" with: Attach volume "clst-west-eu-dev-dynamic-pvc-c209421d-12dd-11e7-af1a-000d3a2735d9.vhd" to instance "k8s-agent-1DA8A8DF-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'clst-west-eu-dev-dynamic-pvc-f843f8fa-0663-11e7-af1a-000d3a2735d9.vhd' to VM 'k8s-agent-1DA8A8DF-2' because the disk is currently being detached. Please wait until the disk is completely detached and then try again."

I can also verify that the Pod is deployed successfully on a different node, right after I delete it and let the ReplicationController reschedule it.

With respect to the "VM where the Pod was originally scheduled", I don't see the Pod being re-scheduled when this error occurs. Here's the event log of a problematic Pod, 20 seconds after I launched it:

Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  20s       20s     1   {default-scheduler }            Normal      Scheduled   Successfully assigned bytecode-ghost-29-rc-tttss to k8s-agent-1da8a8df-2
  20s       3s      6   {controller-manager }           Warning     FailedMount Failed to attach volume "pvc-2e1a5f47-131e-11e7-af1a-000d3a2735d9" on node "k8s-agent-1da8a8df-2" with: Attach volume "clst-west-eu-dev-dynamic-pvc-2e1a5f47-131e-11e7-af1a-000d3a2735d9.vhd" to instance "k8s-agent-1DA8A8DF-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'clst-west-eu-dev-dynamic-pvc-f843f8fa-0663-11e7-af1a-000d3a2735d9.vhd' to VM 'k8s-agent-1DA8A8DF-2' because the disk is currently being detached. Please wait until the disk is completely detached and then try again."

And the full description 10 minutes later:

Name:       bytecode-ghost-29-rc-tttss
Namespace:  default
Node:       k8s-agent-1da8a8df-2/10.240.0.10
Start Time: Mon, 27 Mar 2017 21:50:03 +0300
Labels:     app_name=ghost
        deployment_id=bytecode-ghost-29
        org=bytecode
Status:     Pending
IP:
Controllers:    ReplicationController/bytecode-ghost-29-rc
Containers:
  bytecode-ghost-29:
    Container ID:
    Image:      ghost
    Image ID:
    Port:       2368/TCP
    Limits:
      cpu:  1
      memory:   1Gi
    Requests:
      cpu:      600m
      memory:       768Mi
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Liveness:       http-get http://:2368/ delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:2368/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Volume Mounts:
      /var/lib/ghost from ghost-files (rw)
      /var/lib/ghost/config.js from ghost-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8dmpk (ro)
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  ghost-files:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  bytecode-ghost-29-pvc
    ReadOnly:   false
  ghost-config:
    Type:   ConfigMap (a volume populated by a ConfigMap)
    Name:   bytecode-ghost-29-cm
  default-token-8dmpk:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-8dmpk
QoS Class:  Burstable
Tolerations:    <none>
Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----                -------------   --------    ------      -------
  10m       10m     1   {default-scheduler }                Normal      Scheduled   Successfully assigned bytecode-ghost-29-rc-tttss to k8s-agent-1da8a8df-2
  8m        1m      4   {kubelet k8s-agent-1da8a8df-2}          Warning     FailedMount Unable to mount volumes for pod "bytecode-ghost-29-rc-tttss_default(2ff0e683-131e-11e7-af1a-000d3a2735d9)": timeout expired waiting for volumes to attach/mount for pod "bytecode-ghost-29-rc-tttss"/"default". list of unattached/unmounted volumes=[ghost-files]
  8m        1m      4   {kubelet k8s-agent-1da8a8df-2}          Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "bytecode-ghost-29-rc-tttss"/"default". list of unattached/unmounted volumes=[ghost-files]
  10m       3s      13  {controller-manager }               Warning     FailedMount Failed to attach volume "pvc-2e1a5f47-131e-11e7-af1a-000d3a2735d9" on node "k8s-agent-1da8a8df-2" with: Attach volume "clst-west-eu-dev-dynamic-pvc-2e1a5f47-131e-11e7-af1a-000d3a2735d9.vhd" to instance "k8s-agent-1DA8A8DF-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'clst-west-eu-dev-dynamic-pvc-f843f8fa-0663-11e7-af1a-000d3a2735d9.vhd' to VM 'k8s-agent-1DA8A8DF-2' because the disk is currently being detached. Please wait until the disk is completely detached and then try again."

BTW, I just noticed that all failing Pods I see in my cluster having the same error, try to use the same volume: clst-west-eu-dev-dynamic-pvc-2e1a5f47-131e-11e7-af1a-000d3a2735d9.vhd.

Then I inspected the VM in the Azure Portal and saw this: screenshot 2017-03-27 22 37 24

screenshot 2017-03-27 22 35 07

I'm also attaching an Instance View of the VM where the failing Pods were scheduled (with some data removed): k8s-agent-1DA8A8DF-2-instance-view.json.zip

At this point, I'm not quite sure if this is a k8s issue; not sure what to report to Kubernetes. Looks more like an issue w/ the AzureDisk Volume Plugin I guess?

theobolo commented 7 years ago

Exactly the same problem here !

colemickens commented 7 years ago

What version of Kubernetes are your clusters running?

colemickens commented 7 years ago

I need the instanceView from the machine where the disk was previously attached, otherwise I have no idea what's going on - the error message indicates that the detach never finished from the other node - I need the instanceView to know if that's true.

Also, please let me know the Kubernetes cluster version.

I'm still confused about timeline of events. Are the error messages about "DiskBlobNotFound" occurring at the exact same time as the "failed to attaches"? Is the disk actually detached from wherever it was scheduled? Please review my post above and provide all of the requested information. I can not troubleshoot this any further without it, especially since I can't repro.

This also should be filed as an Issue against Kubernetes repo.

Wait... are you attempting to run multiple pods with the same disk right now???

dbalaouras commented 7 years ago

@colemickens no, I am not attempting to run multiple pods with the same disk. I know this is not possible, plus I need a separate disk mounted to each single Pod and I am using dynamic provisioning.

For each deployment, I create a brand new (with a different name each time) PersistentVolumeClaim using a StorageClass with Provisioner kubernetes.io/azure-disk. See my first post for more details.

After a few deployments, I start seeing this error...I can't be sure in which pod (if any) the disk was originally attached to. During my initial (and quick) investigation, I could not find the disk mentioned in the error msg, mounted to any other node.

With respect to the DiskBlobNotFound error, I noticed that way after I saw the errors in the Pods. So again, can't be sure if that happened before or after the AttachDiskWhileBeingDetached error.

So let me do this maybe: I will try to reproduce in a fresh cluster and try to keep records of the disks that get created and the nodes they get mounted to. In the meanwhile, if you need me to collect any additional info or if you have any hints on what could cause this issue, please let me know.

colemickens commented 7 years ago

"After a few deployments" means what? You spun up a bunch more PVC and Pods and then ran into it? Or you did a rolling update, and the Pod got rescheduled?

If you're in Slack, please ping me, might be able to dig in a bit faster.

dbalaouras commented 7 years ago

@colemickens thx much for the help so far. Posting some updates here to keep this thread synced:

Each "deployment" in my use case includes:

1 ConfigMap

1 PVC

1 RC with a single POD which uses the PVC

1 service

1 Ingress

PVs usually get deleted after I uninstall the deployment (i.e. delete all above resources ^^). I did notice the following tho:

in the Azure portal, several disks (VHDs) with the same URL were shown as mounted to a single VM. That VM was of the 2 Nodes which failed to host any Pod.
I looked into that node and found no VHD mounted.
after "umounting" the disks from the Portal, almost everything got back to normal.

Finally: I am trying to reproduce the issue in a new cluster that runs k8s 1.5.3 but everything seems very stable so far! I will post here any updates.

colemickens commented 7 years ago

Hi @dbalaouras Does this still need to be kept open? If you've not repro'd please close, otherwise give an update. Otherwise I'll go ahead and close this out again. Thanks!

dbalaouras commented 7 years ago

@colemickens sure, I'll close it now and re-open it only if it appears again in 1.5.3. Thanks!

zecke commented 7 years ago

I have this issue right now. The blob is not mounted on any agent but it fails to mount for 8h hours straight.

Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Volumes:
  azure:
    Type:       AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
    DiskName:       DISK-2.vhd
    DiskURI:        https://ACCOUNT.blob.core.windows.net/galera/DISK-2.vhd
    FSType:     ext4
    CachingMode:    ReadWrite
    ReadOnly:       false
  default-token-g95bh:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-g95bh
QoS Class:  Burstable
Tolerations:    <none>
Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----                -------------   --------    ------      -------
  46m       46m     1   {default-scheduler }                Normal      Scheduled   Successfully assigned galera-node3-38x9r to k8s-agent-6dabf06d-0
  45m       1m      23  {controller-manager }               Warning     FailedMount Failed to attach volume "azure" on node "k8s-agent-6dabf06d-0" with: Attach volume "DISK-2.vhd" to instance "k8s-agent-6DABF06D-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'DISK-2.vhd' to VM 'k8s-agent-6DABF06D-0' because the disk is currently being detached. Please wait until the disk is completely detached and then try again."
  44m       33s     21  {kubelet k8s-agent-6dabf06d-0}          Warning     FailedMount Unable to mount volumes for pod "galera-node3-38x9r_default(0dff91b7-23f3-11e7-b279-000d3a28cac3)": timeout expired waiting for volumes to attach/mount for pod "default"/"galera-node3-38x9r". list of unattached/unmounted volumes=[azure]
  44m       33s     21  {kubelet k8s-agent-6dabf06d-0}          Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"galera-node3-38x9r". list of unattached/unmounted volumes=[azure]

colemickens commented 7 years ago

@zecke Would you be able to use https://resources.azure.com to look at the VMs in your cluster. Particularly to look at the instanceView tab for each VM and see if the disk is still attached to one of those VMs?

I'm wondering if a detach operation failed, leaving the disk leased out, but without Kubernetes having any way of knowing that it needs to try to re-detach from the VM? If you do find such a VM, obviously please let us know, but you should be able to mitigate by doing any sort of modification to the VM. Something as simple as adding/removing a tag in the Portal can trigger it to complete the detachment operation.

If this is the case, we can try to get some details from you privately to understand why the detach failed initially. Thank you very much for any information you can provide.

zecke commented 7 years ago

Feel free to ping me out of band, a support request has been filed too. It is a galera cluster with three disks, three k8 agents and three RCs/SVCs. Two disks are attached from the resources view point of view while nothing is mounted on the agents. The specific disk above is not mounted and can't be mounted.

So apparently the storage with k8/Linux is not reliable. What other options do I have?

xyz@k8s-agent-6DABF06D-0:~$ mount | grep sdc
xyz@k8s-agent-6DABF06D-0:~$ 
xyz@k8s-agent-6DABF06D-1:~$ mount | grep sdc
xyz@k8s-agent-6DABF06D-1:~$ 
xyz@k8s-agent-6DABF06D-2:~$ mount | grep sdc
xyz@k8s-agent-6DABF06D-2:~$

zecke commented 7 years ago

Just for the record due out of band communication: The upstream developer(s) have no interest in fixing it. To me it seems if you want kubernetes with persistent storage, then right now don't use Azure. I will not comment on this issue anymore.

rossedman commented 7 years ago

@colemickens @zecke I am having this problem as well. I am using the default StorageClass. I can access the blob store and see that the vhd is created. I can also see the drive is mounted on one of the host nodes under the instanceView mentioned above.

However, when I deploy my pod I get this error:

Failed to attach volume "pvc-67a61ade-2476-11e7-be9d-000d3a18eadc" on node 
"k8s-agent-8492d20d-1" with: Attach volume "<vhd name>.vhd" 
to instance "k8s-agent-8492D20D-1" failed with compute.VirtualMachinesClient#CreateOrUpdate: 
Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with 
status 'Failed': Code="AcquireDiskLeaseFailed" Message="Failed to acquire lease while creating 
disk '<vhd name>.vhd' using blob with URI https://<storage account>.blob.core.windows.net/vhds/<vhd name>.vhd. 
Blob is already in use."

The reason for this is a FailedMount and then followed by a FailedSync. If I try to redeploy this pod it fails because the first deployment is still holding a lease to it or something.

This blob isn't in use. I deleted the PVC, the PV, the Deployment and then the Page Blob from the storage account as well.

Any advice on what to do?

Here are versions I am using as well

Client Version: v1.6.1
Server Version: v1.5.3

colemickens commented 7 years ago

I can assure everyone we're very interested in this being a smooth experience. I apologize for any miscommunication, I simply wanted to ensure the discussion was here on GitHub instead of in personal email. I will personally try to assist with driving this issue.

@zecke Unfortunately, without the information I've requested above, I will be unable to make any further guesses or recommendations without access to your account. If you have opened an Issue with Azure Support, it will eventually make its way to our team where we can potentially look into your subscription on your behalf. If you'd like me to try to assist sooner, please try to collect the requested information from https://resources.azure.com.

@zecke To give a bit more detail on my suspicions... I believe that the disk is being unmounted but not detached. This would explain the behavior you're seeing - the disk is not mounted, but Azure complains that a lease is taken, or that a detach is currently in progress. Unforunately, as I mentioned, I can only confirm this if you collect information from the instanceView. Please let me know if you'd like to look into that, or if you'd like to just have Azure Support drive this discussion going forward.

(Getting your server version would be helpful too. Newer versions have a number of stability improvements.)

colemickens commented 7 years ago

@rossedman Good information, let me ask some followup questions:

What happened after you deleted the PV + page blob? I guess it sounds like you deleted it all at once.
You said you looked at instanceView and saw the disk attached... Was that on the correct VM where the Pod was trying to be launched (k8s-agent-8492d20d-1)? And did it show up in the regular VM GET as well?
How easily/reliably can you repro? Are you willing to work with someone once you have repro'd? (By interactively debugging or granting someone access to the target resource group)

This one is more surprising, I've never had someone have the PV provisioned and then immediately say it's in use.

For everyone's knowledge, there is a large re-write coming for Kubernetes 1.7 that should improve the handling of a variety of edge-cases. That having been said, if we can identify issues in 1.6, we can try to get those fixes as well for future 1.6.x releases.

colemickens commented 7 years ago

And re-opening this since people are clearly having some issues with 1.5.3.

rossedman commented 7 years ago

@colemickens will get some more information today. actually, we are migrating our k8s cluster out of dev/test and into a prod setup so the dev/test cluster we could give access to and try to do some troubleshooting. let me try to do this process again and I will provide answers to the questions above.

My suspicions are that the VHD mount is on the wrong node. The pod may be rescheduling to other nodes due to memory constraints or something else and its staying attached to the original host node. Can't confirm yet but will have more soon.

colemickens commented 7 years ago

@rossedman That is also my suspicion. There is code specifically built to retry the attachment operation, but I fear the same may not exist for detach. (If the detach fails, but isn't specifically handled, when K8s calls to retry the detach, I believe Azure will return immediate success...)

This is why I'm so interested in seeing the instanceView (in @zecke's case as well)...

Thanks!

rossedman commented 7 years ago

@colemickens So here are a couple findings that I have to report:

Files I'm Working With

This is just a PersistentVolumeClaim a single postgres container. Nothing special. I have 3 agents nodes and 1 master. In the instance below the PV mounts to the third host agent.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: sonarqube-vc
  annotations:
    volume.beta.kubernetes.io/storage-class: "default"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sonarqube
  labels:
    app: sonarqube
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      volumes:
      - name: sonarqube-db
        persistentVolumeClaim:
          claimName: sonarqube-vc

      imagePullSecrets:
      - name: acrsecret

      containers:
      - name: postgres
        image: postgres:alpine
        volumeMounts:
        - mountPath: "/var/lib/postgresql/data"
          name: sonarqube-db
        env:
        - name: POSTGRES_USER
          value: whatever
        - name: POSTGRES_PASSWORD
          value: whatever
        - name: POSTGRES_DB
          value: sonarqube
        - name: PGDATA
          value: "/var/lib/postgresql/data/pgdata"
        ports:
        - containerPort: 5432

After the claim is deployed:

kubectl get pvc

NAME           STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
sonarqube-vc   Bound     pvc-0535bb1d-25ea-11e7-86f9-000d3a19ae39   20Gi       RWO           default        17s

kubectl get pv

NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                  STORAGECLASS   REASON    AGE
pvc-0535bb1d-25ea-11e7-86f9-000d3a19ae39   20Gi       RWO           Delete          Bound     default/sonarqube-vc   default                  56s

InstanceView

  "disks": [
    {
      "name": "k8s-agent-2D24F7F6-2-osdisk",
      "statuses": [
        {
          "code": "ProvisioningState/succeeded",
          "level": "Info",
          "displayStatus": "Provisioning succeeded",
          "time": "2017-04-20T16:57:06.3153019+00:00"
        }
      ]
    },
    {
      "name": "absg-prod-acs-build-absg-prod-acs--pvc-0535bb1d-25ea-11e7-86f9-000d3a19ae39.vhd",
      "statuses": [
        {
          "code": "ProvisioningState/succeeded",
          "level": "Info",
          "displayStatus": "Provisioning succeeded",
          "time": "2017-04-20T16:57:06.7840702+00:00"
        }
      ]
    }
  ]

When deploying the above it times out and fails to mount:

If I do a kubectl describe pods/<pod-name> I see that the pod was scheduled on the same k8s agent that the disk is mounted to:

Node:       k8s-agent-2d24f7f6-2/10.240.0.6

Here is the error message from the diskmount:

  32s       32s     1   kubelet, k8s-agent-2d24f7f6-2           Warning     FailedMount Unable to mount volumes for pod "sonarqube-2247042819-v0271_default(63086199-25ea-11e7-86f9-000d3a19ae39)": timeout expired waiting for volumes to attach/mount for pod "default"/"sonarqube-2247042819-v0271". list of unattached/unmounted volumes=[sonarqube-db]
  32s       32s     1   kubelet, k8s-agent-2d24f7f6-2           Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"sonarqube-2247042819-v0271". list of unattached/unmounted volumes=[sonarqube-db]

When deleting a PersistentVolume it does not remove it from the blob storage but does seem to unmount from host node:

If I run kubectl delete pv/<some-generated-name> && kubectl delete pvc/sonarqube-vc this does not result in the VHD being removed from the storage that is in the ACS resource group even thought the ReclaimPolicy is set to Delete. I am using the default StorageClass for this. However, it does not show up in the InstanceView after I do this, which suggests it is not mounted

rossedman commented 7 years ago

@colemickens As a temporary fix I ended up using your kubernetes container that can create VHDs and I mounted that into my container. That process still works.

rossedman commented 7 years ago

@colemickens Hey, wondering if there is any update on this? Was this what you needed? Thanks

ComeMaes commented 7 years ago

Hi everyone, Similar situation here, been troubleshooting it for a while now.

setup info:

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Cluster deployed via ACS.

I have some database server k8s deployments (single pod per deployment, each having their own PVC, dynamically provisioned).

Problems arose after this specific event, as shown by the logs of the kube controller manager:

I0524 00:36:58.826461       1 nodecontroller.go:608] NodeController detected that zone westeurope::1 is now in state FullDisruption.
I0524 00:36:58.826549       1 nodecontroller.go:608] NodeController detected that zone westeurope::2 is now in state FullDisruption.

So judging by the logs that followed, some nodes went down, came back up, k8s rescheduled pods and we have the issue that was documented extensively above.

We tried to force pod deletion in some cases to try to understand what was going on, which resulted in them being recreated (as they're managed by the deployments), and some volumes were correctly reattached. However even this was not always successful, and we have pods that are stuck in the ContainerCreating state:

Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason          Message
  ---------     --------        -----   ----                            -------------   --------        ------          -------
  1h            3s              36      kubelet, k8s-agent-4ba79e32-1                   Warning         FailedMount   Unable to mount volumes for pod "mongodb-deployment-1225361271-d1z50_default(b39cc50b-4098-11e7-beb6-000d3a290a1b)": timeout expired waiting for volumes to
attach/mount for pod "default"/"mongodb-deployment-1225361271-d1z50". list of unattached/unmounted volumes=[mongo-data]
  1h            3s              36      kubelet, k8s-agent-4ba79e32-1                   Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mongodb-deployment-1225361271-d1z50". list of unattached
/unmounted volumes=[mongo-data]

This appears to be the same issue: the blob lease is locked and the volume cannot be attached to the rescheduled pod. The message in the Azure portal VM activity log:

Operation name
Write VirtualMachines
Time stamp
Wed May 24 2017 19:12:23 GMT+0200 (CEST)
Event initiated by
XXXXX-acs-cluster
Error code
AttachDiskWhileBeingDetached
Message
Cannot attach data disk 'XXXXX-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to VM 'k8s-agent-4BA79E32-1' because the disk is currently being detached. Please wait until the disk is completely detached and then try again.

Appears repeatedly, forever:

InstanceView of the VM k8s-agent-4BA79E32-1:

{
  "platformUpdateDomain": 2,
  "platformFaultDomain": 2,
  "vmAgent": {
    "vmAgentVersion": "2.1.3",
    "statuses": [
      {
        "code": "ProvisioningState/succeeded",
        "level": "Info",
        "displayStatus": "Ready",
        "message": "Guest Agent is running",
        "time": "2017-05-24T16:32:23+00:00"
      }
    ],
    "extensionHandlers": [
      {
        "type": "Microsoft.OSTCExtensions.CustomScriptForLinux",
        "typeHandlerVersion": "1.5.2.1",
        "status": {
          "code": "ProvisioningState/succeeded",
          "level": "Info",
          "displayStatus": "Ready",
          "message": "Plugin enabled"
        }
      }
    ]
  },
  "disks": [
    {
      "name": "k8s-agent-4BA79E32-1-osdisk",
      "statuses": [
        {
          "code": "ProvisioningState/succeeded",
          "level": "Info",
          "displayStatus": "Provisioning succeeded",
          "time": "2017-05-24T15:49:53.2215883+00:00"
        }
      ]
    },
    {
      "name": "XXXXX-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd",
      "statuses": [
        {
          "code": "ProvisioningState/deleting",
          "level": "Info",
          "displayStatus": "Deleting"
        }
      ]
    },
    {
      "name": "XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd",
      "statuses": [
        {
          "code": "ProvisioningState/failed/AcquireDiskLeaseFailed",
          "level": "Error",
          "displayStatus": "Provisioning failed",
          "message": "Failed to acquire lease while creating disk 'XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd' using blob with URI https://XXXXXstandardstorage.blob.core.windows.net/vhds/XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd. Blob is already in use.",
          "time": "2017-05-24T15:49:53.518542+00:00"
        }
      ]
    }
  ],
  "extensions": [
    {
      "name": "cse1",
      "type": "Microsoft.OSTCExtensions.CustomScriptForLinux",
      "typeHandlerVersion": "1.5.2.1",
      "statuses": [
        {
          "code": "ProvisioningState/succeeded",
          "level": "Info",
          "displayStatus": "Provisioning succeeded",
          "message": "Command is finished.\n---stdout---\n\n---errout---\n\n"
        }
      ]
    }
  ],
  "statuses": [
    {
      "code": "ProvisioningState/failed/AcquireDiskLeaseFailed",
      "level": "Error",
      "displayStatus": "Provisioning failed",
      "message": "Failed to acquire lease while creating disk 'XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd' using blob with URI https://XXXXXstandardstorage.blob.core.windows.net/vhds/XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd. Blob is already in use.",
      "time": "2017-05-24T15:49:53.5341685+00:00"
    },
    {
      "code": "PowerState/running",
      "level": "Info",
      "displayStatus": "VM running"
    }
  ]
}

So in the instanceView the disk XXXXX-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd is being deleted. The message from the Activity Log of the portal UI says that it cannot be attached because it's being detached... apparently from the same VM.

Finally "statuses" message of the instanceView is the same as the one appearing in the Overview tab of the portal UI :

Provisioning state 
Provisioning failed. Failed to acquire lease while creating disk 'XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd' using blob with URI https://XXXXXstandardstorage.blob.core.windows.net/vhds/XXXXX-dynamic-pvc-eeaf97ab-25e6-11e7-beb6-000d3a290a1b.vhd. Blob is already in use.. AcquireDiskLeaseFailed

Which refers to another disk that is actually bound to another pod running in another instance without any problem. I don't understand why that node would be trying to acquire the lease of this volume...

This is a heavy and unexpected blocker in the process of moving to k8s on Azure for us. Happy to discuss and help troubleshooting this further (slack, email, on here or wherever), as it's a critical point to elude before moving forward with Azure. @colemickens Please let me know.

dbalaouras commented 7 years ago

@colemickens sure, I'll close it now and re-open it only if it appears again in 1.5.3. Thanks!

@colemickens , unfortunately, this issue came back to our v1.5.3 cluster as well; I'm switching to Azure File to see if that works better.

andrey-moor commented 7 years ago

Same thing happens on 1.6.2...

ams0 commented 7 years ago

I'm looking into this as well, and on a 1.5.3 ACS-RP deployed cluster I can't reproduce the issue. I can move pods+volume back and forth the two nodes in my cluster 100% of the times; indeed, there's a bunch of errors both in the pod and in the controller, but on average after 5-6 minutes the pod is in running state:

At 22:38:16 I kill my pod:

I0607 22:38:16.303229       1 replication_controller.go:322] Observed updated replication controller nginxrc. Desired pod count change: 1->1

From 22:46:02 to 22:46:33 it tries to detach the volume, unsuccessfully:

E0607 22:46:33.271134       1 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd\"" failed. No retries permitted until 2017-06-07 22:46:33.771106254 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-27aae708-4b90-11e7-8850-000d3a20df89" on node "k8s-agent-6568816c-0" with: Attach volume "kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" to instance "k8s-agent-6568816C-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with status 'Failed': Code="AcquireDiskLeaseFailed" Message="Failed to acquire lease while creating disk 'kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd' using blob with URI https://kubedisk.blob.core.windows.net/vhds/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd. Blob is already in use."

but moments later the attach/detach operations succeeds:

I0607 22:46:33.803447       1 reconciler.go:178] Started DetachVolume for volume "kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" from node "k8s-agent-6568816c-1"
I0607 22:46:33.810071       1 operation_executor.go:754] Verified volume is safe to detach for volume "kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" (spec.Name: "pvc-27aae708-4b90-11e7-8850-000d3a20df89") from node "k8s-agent-6568816c-1".
I0607 22:46:51.397015       1 attacher.go:146] VolumesAreAttached: check volume "kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" (specName: "pvc-27aae708-4b90-11e7-8850-000d3a20df89") is no longer attached
I0607 22:46:51.397061       1 operation_executor.go:565] VerifyVolumesAreAttached determined volume "kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" (spec.Name: "pvc-27aae708-4b90-11e7-8850-000d3a20df89") is no longer attached to node %!q(MISSING), therefore it was marked as detached.
I0607 22:47:29.614566       1 operation_executor.go:700] DetachVolume.Detach succeeded for volume "kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" (spec.Name: "pvc-27aae708-4b90-11e7-8850-000d3a20df89") from node "k8s-agent-6568816c-1".
I0607 22:47:29.672353       1 reconciler.go:213] Started AttachVolume for volume "kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" to node "k8s-agent-6568816c-0"
I0607 22:48:15.389463       1 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/azure-disk/kubedisk-dynamic-pvc-27aae708-4b90-11e7-8850-000d3a20df89.vhd" (spec.Name: "pvc-27aae708-4b90-11e7-8850-000d3a20df89") from node "k8s-agent-6568816c-0".

and the pod is happy again. So I think the observed issue is due to some kind of race condition, but I fail to reproduce it in my simple setup.

P.S.

@ComeMaes I did indeed get once:

I0607 22:48:25.251253       1 nodecontroller.go:608] NodeController detected that zone westeurope::1 is now in state FullDisruption.

which had my kube-dns and a bunch more pods restarted, but I don't think it's related to the disk provisioning.

rossedman commented 7 years ago

@ams0 I will give this a shot again since we upgraded to 1.6.2 but had gone through this process quite a few times. Redeployed with acs-engine and will see how it works with that as well. Thanks for the update. Will report any findings.

ComeMaes commented 7 years ago

@ams0 Actually, looking at the controller logs I have seen that same FullDisruption message quite a few times but that did not lead to the situation I described above. the volumes/pods assignations are shuffled around but seem to resolve to a stable status after a few minutes. So I am pretty clueless about what specifically differed and made things go wrong...

btw, that message does not seem to match Azure's status reports, but that's another issue.

ryannix123 commented 7 years ago

I'm starting to see this as well:

Failed to restart the virtual machine 'nuopenshift-node-1'. Error: Failed to acquire lease while creating disk 'kubernetes-dynamic-pvc-3d1f8df5-3f01-11e7-861a-000d3a029a5b.vhd' using blob with URI https://pv1sa2xen6xyzu3dgs.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-3d1f8df5-3f01-11e7-861a-000d3a029a5b.vhd. Blob is already in use.

I can't stop or restart the node. I'm using OpenShift Origin 3.5.

mtbbiker commented 7 years ago

I have the same issue, I have deployed Kafka a statefulset using pvs (K8S 1.5.3) Initially the deployment was sucessful and now I see the same issue as described above. Failed to attach volume "pvc-17ae0940-4db2-11e7-93a6-000d3a28d311" on node "k8s-agent-c07fe190-3" with: Attach volume "k8sansiot00-dynamic-pvc-17ae0940-4db2-11e7-93a6-000d3a28d311.vhd" to instance "k8s-agent-C07FE190-3" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'k8sansiot00-dynamic-pvc-7559901a-4dab-11e7-93a6-000d3a28d311.vhd' to VM 'k8s-agent-C07FE190-3' because the disk is currently being detached. Please wait until the disk is completely detached and then try again."

mtbbiker commented 7 years ago

Some feedback: The only way I could get my statefulset back to work was to do the following

Delete the statefulset with kubectl
Delete pvc an pv in the statefulset's namespace
Through the azure portal unmount the containers associate with the storage.
Redeploy

rossedman commented 7 years ago

@mtbbiker I had to go through this quite a few times as well. Sometimes even forcing an unmount on the host node.

ryannix123 commented 7 years ago

@mtbbiker thanks for the listing the fix. Would you mind posting the steps in more detail? Thank you.

mtbbiker commented 7 years ago

@ryannix123 I will post them here

rossedman commented 7 years ago

@ryannix123 @mtbbiker how is this a fix you are deleting the persistent volume? I am confused as to what this solves.

ryannix123 commented 7 years ago

@rossedman In my case I wonder if it's an old PV claim that is causing problems. We're running OpenShift on Azure as a pilot right now so there isn't much on the cluster.

mtbbiker commented 7 years ago

@rossedman Kubernetes were showing the Pods in a Pending state, it was left in that state for more than 20 min, in my book not acceptable on a Pod failure, i.e the Pods just didn't start again as it was complaining about not being able to mount the storage. I basically had to delete the Kafka Statefull set (3 replicas) and clean up the deployment. I then had to unmount the "orphan" storage containers in Azure (Did this through the azure portal), Delete the containers in Azure. I then had to deploy the Stateful set again. If there is interest I will when back in the office, post the yaml file and links of details to my Kafka deployment.

andrey-moor commented 7 years ago

@colemickens is there any solution for that? I've seen it in both 1.5 and 1.6.

rocketraman commented 7 years ago

Just encountered this same problem on a test cluster, 1.5.3, built from acs-engine. Seemed to occur after a node went down. Even having a good workaround would be useful.

hieuluong commented 7 years ago

Everybody, I have a idea, Can we use NFS as a server ? How do you think about this ?

dbalaouras commented 7 years ago

@rocketraman @hieuluong Just FYI, we moved away from Azure Disk and currently using Azure File (SMB-based). $MS has put together a blog post comparing the two.

I'd probably recommend Azure File for a few reasons:

created and mounted very fast
you can browse the files stored within from the Azure Portal
no performance degradation compared to Azure Disk (as far as I see)
there's no Azure limit on the number of mounted Azure Files. (Azure Disks are limited to [number_of_cores x 2] in most cases which was a no-go for us; see Azure VM Sizes.

One downside is that there's no Azure Files snapshots, which are available on Azure Disks; I had to put together a small tool to help me w/ Azure File backups.

rocketraman commented 7 years ago

@dbalaouras That blog post is interesting. According to it, performance for File mode is equivalent to or better than Disk for both throughput and iops:

Max 8KB IOps : 500 IOps disk vs 1000 IOps File
Throughput : Up to 60 MB/s per Disk for both Disk and File

Surprising! Has anyone confirmed these numbers with real-world usage?

discordianfish commented 7 years ago

First of all, I'm using vanilla k8s. Just answering here because there were no reports in upstream. Can create one though if wanted. I just ran into the same/similar? issue. My PVC is here:

data-prometheus-1        Bound     pvc-xx-5b49-11e7-9ebf-yy   20Gi       RWO           standard       22h

But my pod won't come up because:

  22h       39s     658 attachdetach                Warning     FailedMount Failed to attach volume "pvc-04b504b2-5b49-11e7-9ebf-yy" on node "xx-worker2" with: Attach volume "kubernetes-dynamic-pvc-xx-5b49-11e7-9ebf-000d3a2a8f26.vhd" to instance "xx-worker2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk 'kubernetes-dynamic-pvc-xx-5b17-11e7-9ebf-yy.vhd' to VM 'xx-worker2' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again."

In the Azure portal it shows 'Failed` with the following (pretty much unformatted) text:

Provisioning state Provisioning failed. Unable to find VHD blob with URI https://70a3e3c6.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-xx-5b17-11e7-9ebf-yy.vhd for disk 'kubernetes-dynamic-pvc-xx-5b17-11e7-9ebf-yy.vhd'.. DiskBlobNotFound
...
kubernetes-dynamic-pvc-8bc14169-5b17-11e7-9ebf-000d3a2a8f26.vhd Provisioning failed. Unable to find VHD blob with URI https://70a3e3c6.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-xx-5b17-11e7-9ebf-yy.vhd for disk 'kubernetes-dynamic-pvc-xx-5b17-11e7-9ebf-yy.vhd'.. DiskBlobNotFound

Here the relevant bits from instanceView:

{
  "bootDiagnostics": null,
  "disks": [
    {
      "name": "worker-osdisk",
      "statuses": [
        {
          "code": "ProvisioningState/succeeded",
          "displayStatus": "Provisioning succeeded",
          "level": "Info",
          "message": null,
          "time": "2017-06-27T09:14:34.806528+00:00"
        }
      ]
    },
    {
      "name": "kubernetes-dynamic-pvc-xx-5b17-11e7-9ebf-yy.vhd",
      "statuses": [
        {
          "code": "ProvisioningState/deleting",
          "displayStatus": "Deleting",
          "level": "Info",
          "message": null,
          "time": null
        }
      ]
    },
    {
      "name": "kubernetes-dynamic-pvc-zz-5b17-11e7-9ebf-yy.vhd",
      "statuses": [
        {
          "code": "ProvisioningState/succeeded",
          "displayStatus": "Provisioning succeeded",
          "level": "Info",
          "message": null,
          "time": "2017-06-27T09:14:34.806528+00:00"
        }
      ]
    },
    {
      "name": "kubernetes-dynamic-pvc-nn-5b17-11e7-9ebf-yy.vhd",
      "statuses": [
        {
          "code": "ProvisioningState/failed/DiskBlobNotFound",
          "displayStatus": "Provisioning failed",
          "level": "Error",
          "message": "Unable to find VHD blob with URI https://70a3e3c6.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-nn-5b17-11e7-9ebf-yy.vhd for disk 'kubernetes-dynamic-pvc-nn-5b17-11e7-9ebf-yy.vhd'.",
          "time": "2017-06-27T09:14:34.837783+00:00"
        }
      ]
    }
  ],
  "extensions": null,
  "platformFaultDomain": 2,
  "platformUpdateDomain": 2,
  "rdpThumbPrint": null,
  "statuses": [
    {
      "code": "ProvisioningState/failed/DiskBlobNotFound",
      "displayStatus": "Provisioning failed",
      "level": "Error",
      "message": "Unable to find VHD blob with URI https://70a3e3c6.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-nn-5b17-11e7-9ebf-yy.vhd for disk 'kubernetes-dynamic-pvc-nn-5b17-11e7-9ebf-yy.vhd'.",
      "time": "2017-06-27T09:14:34.837783+00:00"
    },
    {
      "code": "PowerState/running",
      "displayStatus": "VM running",
      "level": "Info",
      "message": null,
      "time": null
    }
  ],
  "vmAgent": {
    "extensionHandlers": [],
    "statuses": [
      {
        "code": "ProvisioningState/succeeded",
        "displayStatus": "Ready",
        "level": "Info",
        "message": "Guest Agent is running",
        "time": "2017-06-28T13:12:31+00:00"
      }
    ],
    "vmAgentVersion": "2.2.13"
  }
}

This happened on the initial deployment, so no disks exited before I deployed the stateful set. But it worked before and nothing changed since then, therefor I assume some race condition. If I remove the PVC and pod and start over, the pod gets created on the same system as before and I run into the same error. After condoning that host and reschedule the pod, the disk gets attached and the pods come up.

hieuluong commented 7 years ago

Azure upgraded kubectl to 1.6.6. Now, when you create new ACS, the default verions of kubernetes is 1.6.6.

rossedman commented 7 years ago

Can confirm the 1.6.6 updates fixed this problem for me.

rocketraman commented 7 years ago

I've only been running 1.6.6 for a week, but agreed: so far I haven't had any issues either.

discordianfish commented 7 years ago

To me it happend on vanilla k8s 1.6.6. Was there something else introduced in ACS?

rossedman commented 7 years ago

@discordianfish ACS supports 1.6.6 right now. I believe the ACS team made some adjustments to their implementation of the Cloud Controller Interface and it got rolled out with that release. There was some stuff that was stuck in merge conflict world for awhile on the k8s side I think too. Either way, excited to have this working.

I think we can call this closed? Thanks @colemickens and team.

Azure / ACS