search

remche / terraform-openstack-rke

Terraform Openstack RKE
Mozilla Public License 2.0
37 stars 20 forks source link

Creating PVs fails when Openstack has multiple compute AZs, but only one storage AZ #74

Open Linutux opened 4 years ago

Linutux commented 4 years ago

Creating PVs fails, wehen installing in an openstack environment, where you have multiple AZs for compute (e.g. ams-a, ams-b, ams-c) but only one for storage (e.g. nova). Kubernetes tries to create them in the compute AZs, which openstack doesn't allow.

There is a solution in https://github.com/openshift/installer/issues/2844#issuecomment-569231621

Especially you need the "ignore_volume_az" option in cloud provider config.

Linutux commented 4 years ago

This config value has to be set: https://registry.terraform.io/providers/rancher/rke/latest/docs/resources/cluster#ignore_volume_az

remche commented 4 years ago

Option added in v0.5.4 release, can you confirm it's working as desired ?

Linutux commented 4 years ago

Option added in v0.5.4 release, can you confirm it's working as desired ?

No, its not working:

Failed to provision volume with StorageClass "cinder": failed to create a 19 GB volume: Bad request with: [POST https://volume.api.ams.fuga.cloud:443/v3/922d27f06e9a42549509b420ff427fc0/volumes], error message: {"badRequest": {"message": "Availability zone 'ams-b' is invalid.", "code": 400}}

Linutux commented 4 years ago

Ok, I think I found out why it is not working:

According to https://kubernetes.io/docs/concepts/storage/storage-classes/#openstack-cinder you need to specify

parameters: availability: nova

when creating the storage class for cinder, otherwise ...

availability: Availability Zone. If not specified, volumes are generally round-robin-ed across all active zones where Kubernetes cluster has a node.

So, in my case the node (Openstack compute) AZs are ams-a, ams-b, ams-c. But the storage AZ is ams. But instead of using ams, the storage class tries to create one in ams-a, -b or -c, unless the parameter "availabilty" is set.

unfortunately I don't see any option to control provider_rke to create this option when setting up the cluster.

Edit: Yes, after creating a new cinder storage class manually, everything is working now.

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: cinder provisioner: kubernetes.io/cinder parameters: availability: ams

remche commented 4 years ago

@Linutux the storage classes are created here. It should not be too difficult to add a variable for avaibility zone. I wonder if it should be set by storage_types or globally.

Linutux commented 4 years ago

I think it should be set by storage_types. Maybe you have SSDs in DC1 and DC2, but only HDDs in DC3 and DC4