search

vexxhost / magnum-cluster-api

Cluster API driver for OpenStack Magnum
Apache License 2.0
47 stars 22 forks source link

feature: add option to allow cinder CSI availability zone override #366

Open satishdotpatel opened 6 months ago

satishdotpatel commented 6 months ago

I have my worker nodes and cinder storage running on different availability zone and because of that getting affinity issue during pv and deployment. By default during cluster create k8s and CSI use same AZ. I would like to have option available where we can tell mcap to override CSI AZ to be different from worker/compute AZ. (Example: csi_availability_zone label apply to template or cluster create)

As you can see in following output topology.cinder.csi.openstack.org/zone=general and topology.kubernetes.io/zone=general is same AZ but I would like to change csi AZ to something else. 

(venv-openstack) root@os-eng-ctrl-01:/tmp# kubectl describe node kube-acrsf-default-worker-rxmg6-t7wzd
Name:               kube-acrsf-default-worker-rxmg6-t7wzd
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=gen.c4-m8-d40
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=Boston-eng
                    failure-domain.beta.kubernetes.io/zone=general
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kube-acrsf-default-worker-rxmg6-t7wzd
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/worker=
                    node.cluster.x-k8s.io/nodegroup=default-worker
                    node.kubernetes.io/instance-type=gen.c4-m8-d40
                    topology.cinder.csi.openstack.org/zone=general
                    topology.kubernetes.io/region=Boston-eng
                    topology.kubernetes.io/zone=general
mnaser commented 6 months ago

I think this is a two part issue:

StorageClass not including the correct mapping By default, it doesn't include anything about the availability zones which the volume is available at -- making it not possible to create the volume in certain situations. We should include this data by including the following to the StorageClass:

...
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - nova
...

Allow cross-AZ attachments So while the above can be sorted, the other issue is inside an environment where cross_az_attach is set to true in the cloud, this will mean that we need to enable ignore-volume-az inside the BlockStorage if the user requires this. This would be handled by adding [BlockStorage]/ignore-volume-az to true in the Cinder CSI plugin configuration file.

satishdotpatel commented 6 months ago

I don't think the allowedTopologies approach will work for this case. But I might consider it a bug in csi-cinder that it doesn't give you a way to set topology.cinder.csi.openstack.org/zone correctly on the nodes/worker.

satishdotpatel commented 6 months ago

How to tell end users to set [BlockStorage]/ignore-volume-az in your k8s cluster. Are there any way to set this during k8s cluster create using magnum?

mnaser commented 6 months ago

I don't think the allowedTopologies approach will work for this case. But I might consider it a bug in csi-cinder that it doesn't give you a way to set topology.cinder.csi.openstack.org/zone correctly on the nodes/worker.

I am not following here.. Cinder CSI doesn't know anything about your topology, so in order for it to understand the topology, we need to provide that information to it.

How to tell end users to set [BlockStorage]/ignore-volume-az in your k8s cluster. Are there any way to set this during k8s cluster create using magnum?

I think for this one, you'll have to probably edit the CCM configuration (which I can't remember if it was a configmap or inside the /etc/kubernetes file).