PVC is not creating all PVs with azure disk across the zones

[PrabhuMathi]

What happened:

Trying to create a statefulset with PVC azure disk. When the pod is created, PV is getting created with azure disks 98% in zone 1 and 2% is other zones. Did some research and tried labeling storage class label with "failure-domain.beta.kubernetes.io/zone: "1 2 3" still not helping. Any suggestions will help, thanks advance.

What you expected to happen:

Expectation is to get the disk to be created across the zone, so it will get attached to existing VMs in different zones

How to reproduce it (as minimally and precisely as possible): Just create a statefulset pods in a AKS cluster enable with "--zone 1 2 3" and statefulset PV mostly create in zone 1 and very minimal in other zones

In my case 98% in zone 1 and 2% zone 2

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.16.9
• Size of cluster (how many worker nodes are in the cluster?): 10
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.): micorservices, Java & IBM messaging queue
• Others:

[andyzhangx]

disk in zone#1 cannot be attached to disk in zone#2, that's a restriction from azure disk. You could add volumeBindingMode: WaitForFirstConsumer in azure disk storage class, by that way, the disk will be created after pod is scheduled to the node, and disk will be created in that zone. While again disk in zone#1 cannot be attached to disk in zone#2, that's a restriction from azure disk.

Here is an example: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/storageclass-azuredisk-csi-topology.yaml#L10

[andyzhangx]

let me know if you have any question

[PrabhuMathi]

@andyzhangx Thanks for your response.

The solution you have suggested is an workaround to get the pod assignment 1st and disk created replicating the VM's Zone.

I'm aware disk in zone 1 cannot get attached in VM in different zone.

Problem I'm facing is whenever a disk is requested by PV, azure is creating the disk is in zone 1. This happens 98% of the times. This happening in storage class config with "volumeBindingMode: Immediate" event with label "failure-domain.beta.kubernetes.io/zone: "1 2 3". This is an issue right?

Please let me know if above make sense.

[andyzhangx]

@andyzhangx Thanks for your response. I'm aware disk in zone 1 cannot get attached in VM in different zone. Problem I'm facing is whenever a disk is requested in PVC azure is creating the disk is in zone 1. This happens 98% of the time when I tried.

@PrabhuMathi can you try add volumeBindingMode: WaitForFirstConsumer in azure disk storage class? thanks.

[PrabhuMathi]

@andyzhangx Thanks for your response. I'm aware disk in zone 1 cannot get attached in VM in different zone. Problem I'm facing is whenever a disk is requested in PVC azure is creating the disk is in zone 1. This happens 98% of the time when I tried.

@PrabhuMathi can you try add volumeBindingMode: WaitForFirstConsumer in azure disk storage class? thanks.

Hi @andyzhangx It worked, but my question "why is disk is requested by PV, azure is creating the disk is in zone 1. This happens 98% of the times. This happening in storage class config with "volumeBindingMode: Immediate" event with label "failure-domain.beta.kubernetes.io/zone: "1 2 3" "

[andyzhangx]

I guess the pv-controller always tries to schedule the pod to the first node, the PV zone value depends on selectedNode: https://github.com/kubernetes/kubernetes/blob/73fa63a86d1d47d82b435ca85c2f350da06b08b9/pkg/volume/azure_dd/azure_provision.go#L253