search

kubernetes-sigs / azuredisk-csi-driver

Azure Disk CSI Driver
Apache License 2.0
144 stars 183 forks source link

Azure Disk in a Multi Region Kubernetes cluster is provisioned in the wrong Region #2343

Open MrAbaddon opened 3 months ago

MrAbaddon commented 3 months ago

What happened:

I have a self managed cluster in Azure in 2 different regions. North Europe and west Europe. 1 control plane and 3 worker nodes in North Europe. 1 control plane and 3 worker nodes in West Europe. I apply a PVC manifest and a Statefullset with node selector for westeurope and I see that the azure disk is provisioned in northeurope.

cat <<EOF | kubectl apply --filename=-
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-azuredisk
  namespace: default
  labels:
    app: nginx
spec:
  podManagementPolicy: Parallel  # default is OrderedReady
  serviceName: statefulset-azuredisk
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/dedicatedhost: host-02 ### host-02 is an azure dedicated host with 3VM for worker nodes in northeurope
      containers:
        - name: statefulset-azuredisk
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/azuredisk/outfile; sleep 1; done
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/azuredisk
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
      spec:
        storageClassName: managed-csi
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
EOF

Looking at the logs in the controller:

I0528 21:30:38.162910       1 utils.go:105] GRPC call: /csi.v1.Controller/CreateVolume
I0528 21:30:38.162962       1 utils.go:106] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","parameters":{"csi.storage.k8s.io/pv/name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0528 21:30:38.163158       1 controllerserver.go:219] begin to create azure disk(pvc-669b1eae-d04b-485a-9987-8f95d86b2a10) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone() maxShares(0)
I0528 21:30:38.163177       1 azure_managedDiskController.go:111] azureDisk - creating new managed Name:pvc-669b1eae-d04b-485a-9987-8f95d86b2a10 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:30:40.948927       1 azure_managedDiskController.go:304] azureDisk - created new MD Name:pvc-669b1eae-d04b-485a-9987-8f95d86b2a10 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:30:40.948954       1 controllerserver.go:317] create azure disk(pvc-669b1eae-d04b-485a-9987-8f95d86b2a10) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-669b1eae-d04b-485a-9987-8f95d86b2a10 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:default]) successfully
I0528 21:30:40.949006       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=2.785803836 request="azuredisk_csi_driver_controller_create_volume" resource_group="xxxxxxxxx" subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" source="disk.csi.azure.com" volumeid="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-669b1eae-d04b-485a-9987-8f95d86b2a10" result_code="succeeded"
I0528 21:30:40.949017       1 utils.go:112] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-669b1eae-d04b-485a-9987-8f95d86b2a10"}}

We can see that in the Request it detects the topology"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1". Thats correct. But the controller then creates in location(northeurope) size(10) diskZone(). northeurope without an AZ and this is wrong.

If I change the node selector to northeurope, the azure disk is provisioned in northeurope and everything works good.

cat <<EOF | kubectl apply --filename=-
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-azuredisk
  namespace: default
  labels:
    app: nginx
spec:
  podManagementPolicy: Parallel  # default is OrderedReady
  serviceName: statefulset-azuredisk
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/dedicatedhost: host-01 ### host-01 is an azure dedicated host with 3VM for worker nodes in westeurope
      containers:
        - name: statefulset-azuredisk
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/azuredisk/outfile; sleep 1; done
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/azuredisk
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
      spec:
        storageClassName: managed-csi
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
EOF

And the Logs:

I0528 21:38:38.132216       1 utils.go:105] GRPC call: /csi.v1.Controller/CreateVolume
I0528 21:38:38.132724       1 utils.go:106] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"northeurope-1","topology.kubernetes.io/zone":"northeurope-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"northeurope-1","topology.kubernetes.io/zone":"northeurope-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","parameters":{"csi.storage.k8s.io/pv/name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"blockchain","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0528 21:38:38.132827       1 controllerserver.go:219] begin to create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone(northeurope-1) maxShares(0)
I0528 21:38:38.132839       1 azure_managedDiskController.go:111] azureDisk - creating new managed Name:pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:38:41.020827       1 azure_managedDiskController.go:304] azureDisk - created new MD Name:pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:38:41.020855       1 controllerserver.go:317] create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:blockchain]) successfully
I0528 21:38:41.020879       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=2.888026849 request="azuredisk_csi_driver_controller_create_volume" resource_group="xxxxxxxxx" subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" source="disk.csi.azure.com" volumeid="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60" result_code="succeeded"
I0528 21:38:41.020890       1 utils.go:112] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"northeurope-1"}}],"capacity_bytes":10737418240,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"blockchain","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60"}}
I0528 21:38:42.288867       1 utils.go:105] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0528 21:38:42.289473       1 utils.go:106] GRPC request: {"node_id":"xxxxxxxxx-host-01-worker-02","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"blockchain","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1716913765467-4400-disk.csi.azure.com"},"volume_id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60"}
I0528 21:38:42.795078       1 controllerserver.go:429] GetDiskLun returned: cannot find Lun for disk pvc-6083de42-5d50-4b6b-84ca-2812374d9a60. Initiating attaching volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 to node xxxxxxxxx-host-01-worker-02 (vmState Succeeded).
I0528 21:38:42.795103       1 controllerserver.go:456] Trying to attach volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 to node xxxxxxxxx-host-01-worker-02
I0528 21:38:42.795115       1 azure_controller_common.go:212] wait 1000ms for more requests on node xxxxxxxxx-host-01-worker-02, current disk attach: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60
I0528 21:38:43.795315       1 azure_controller_common.go:226] Trying to attach volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 lun 0 to node xxxxxxxxx-host-01-worker-02, diskMap len:1, map[/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60:0xc000207e80]
I0528 21:38:43.795346       1 azure_controller_standard.go:103] azureDisk - update(xxxxxxxxx): vm(xxxxxxxxx-host-01-worker-02) - attach disk list(map[/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60:0xc000207e80])
I0528 21:38:44.440915       1 azure_controller_standard.go:116] azureDisk - update(xxxxxxxxx): vm(xxxxxxxxx-host-01-worker-02) - attach disk list(map[/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60:0xc000207e80]) returned with <nil>
I0528 21:38:48.663923       1 azure_controller_standard.go:126] DeleteCacheForNode(xxxxxxxxx-host-01-worker-02) successfully
I0528 21:38:48.663941       1 azure_controller_standard.go:269] updateCache(xxxxxxxxx-host-01-worker-02) successfully
I0528 21:38:48.664012       1 azure_controller_common.go:504] azureDisk - find disk: lun 0 name pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 uri /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60
I0528 21:38:48.664027       1 controllerserver.go:465] Attach operation successful: volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 attached to node xxxxxxxxx-host-01-worker-02.
I0528 21:38:48.664037       1 controllerserver.go:489] attach volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 to node xxxxxxxxx-host-01-worker-02 successfully
I0528 21:38:48.664056       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=6.157552193 request="azuredisk_csi_driver_controller_publish_volume" resource_group="xxxxxxxxx" subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" source="disk.csi.azure.com" volumeid="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60" node="xxxxxxxxx-host-01-worker-02" result_code="succeeded"
I0528 21:38:48.664064       1 utils.go:112] GRPC response: {"publish_context":{"LUN":"0"}}

We can see that in request it detects topology "topology.disk.csi.azure.com/zone":"northeurope-1","topology.kubernetes.io/zone":"northeurope-1". Thats correct. And the controller creates in correct location: begin to create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone(northeurope-1) maxShares(0).

What you expected to happen:

The correct azure disk is provisioned in the correct region and zone

Environment:

andyzhangx commented 3 months ago

if you set volumeBindingMode: WaitForFirstConsumer in storage class, the pod would be scheduled to the node first, and then pvc creation would happen, that could make sure you have the right region together with pod. Otherwise how does the disk controller know which region you want to create a disk pv?

MrAbaddon commented 2 months ago

This is my StorageClasse manifest.

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-csi annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: disk.csi.azure.com parameters: skuName: StandardSSD_LRS # available values: StandardSSD_LRS, StandardSSD_ZRS, Premium_LRS, Premium_ZRS, etc. reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

So I've seted volumeBindingMode: WaitForFirstConsumer. I can see in logs that the request is made for the correct region:

I0528 21:30:38.162962 1 utils.go:106] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","parameters":{"csi.storage.k8s.io/pv/name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}

The pod is created in a node in westeurope but the disk is requested where the controller is based, in this case northeurope:

I0528 21:38:38.132827 1 controllerserver.go:219] begin to create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone(northeurope-1) maxShares(0)

Im using the helm charts to install the driver. So as a work around I'm using the values to force the controller to be replica 1 and with node selectors I'm placing the controller in a node that is located in a region where I want to place the disk. A bit manual but works...