search

kubernetes / cloud-provider-openstack

Apache License 2.0
623 stars 611 forks source link

[manila-csi-plugin] add support for share group and affinity annotations #2685

Closed kayrus closed 1 week ago

kayrus commented 1 month ago

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:

The Manila API allows to specify a share group in which the new share should be created. Along with this it supports affinity/anti-affinity scheduler hints.

What you expected to happen:

The share group ID can be specified in manila storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-manila-nfs
provisioner: nfs.manila.csi.openstack.org
allowVolumeExpansion: true
parameters:
  # Manila share type
  type: default
  # Manila share group
  group-id: %my-group-id%

The scheduler hints for a particular share can be specified in a PVC annotation like it's donw in AWS EBS CSI driver:

$ k describe pv           
Name:              pvc-6bff47f9-1843-4576-ba15-158c73491e8c
Labels:            <none>
Annotations:       ebs.csi.aws.com/iops: 4000
                   ebs.csi.aws.com/volumeType: io2

The proposed annotation keys are:

The PVC with the annotations below:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    manila.csi.openstack.org/affinity: a07a01c7-43b9-484a-8745-717e44d8f662,749ad593-0a99-4e32-91ba-7106db7ef792
    manila.csi.openstack.org/anti-affinity: 364f7c8c-7a60-4023-8458-b332e176789a,1101d1d5-b68d-4f17-8604-46271852b6db
    manila.csi.openstack.org/group-id: 5b5f4dce-6437-461e-80bb-77b0a1f3671f

and storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-manila-nfs
provisioner: nfs.manila.csi.openstack.org
allowVolumeExpansion: true
parameters:
  type: default
  group-id: ca8e84aa-a0c7-4631-9b82-69c95b50a4bd

will produce an API call below:

{
    "share": {
        "share_proto": "nfs",
        "share_network_id": "713df749-aac0-4a54-af52-10f6c991e80c",
        "share_group_id": "5b5f4dce-6437-461e-80bb-77b0a1f3671f",
        "name": "pvc_123",
        "size": 1,
        "scheduler_hints": {
            "same_host": "a07a01c7-43b9-484a-8745-717e44d8f662,749ad593-0a99-4e32-91ba-7106db7ef792",
            "different_host": "364f7c8c-7a60-4023-8458-b332e176789a,1101d1d5-b68d-4f17-8604-46271852b6db"
        }
    }
}

Accessing PVC annotations can be supported with a side-car modifier, see https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1600 for example.

How to reproduce it:

n/a

Anything else we need to know?:

Upstream gophercloud feature request for the scheduler hints: https://github.com/gophercloud/gophercloud/issues/3204

Additionally we can implement the VolumeAttributesClass support. VolumeAttributesClass parameters will override StorageClass parameters, and PVC annotations will override VolumeAttributesClass parameters. For the volumeAttributesClassName support manila CSI must have a k8s client configured. However this can be an extra task. See https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1941 for reference. This approach won't require a side-car container to fetch PVC annotations.

Environment:

kayrus commented 1 month ago

I reviewed the https://github.com/awslabs/volume-modifier-for-k8s sidecar code, and it seems there's no straightforward way to intercept the CreateVolume request without resorting to workarounds like mutexKV[volID]. When PVC annotations feature is enabled, the CreateVolume request must wait for the corresponding ModifyVolumeProperties request to be processed, which introduces complexity.

An alternative approach would be to embed a Kubernetes client inside the CSI controller and fetch annotations during the CreateVolume request.

Ideally, leveraging VolumeAttributesClass parameters would be a cleaner solution. However, due to affinity constraints (like the volume ID list), this parameter would likely be unique per PVC rather than a shared VolumeAttributesClass.