Adding PVC's annotation to CreateVolumeRequest.Parameters

orainxiong commented 6 years ago

As the title says, there is a great possibility that the application-specific Operator would leverage PVC's Annotation for storing custom data. The custom data will tell storage provisioner how CSI drive create volume correctly.

Here is already existing logic for provision volume :

    // Create a CSI CreateVolumeRequest and Response
    req := csi.CreateVolumeRequest{

        Name:       pvName,
        Parameters: options.Parameters,
        VolumeCapabilities: []*csi.VolumeCapability{
            {
                AccessType: accessType,
                AccessMode: accessMode,
            },
        },
        CapacityRange: &csi.CapacityRange{
            RequiredBytes: int64(volSizeBytes),
        },
    }

Note : more details

CSI Plugin specifies that CreateVolumeRequest's field Parameters is OPTIONAL and opaque. What I am suggesting is that we could pass in PVC's Annotation in addition to Storageclass.Parameter.

clintonk commented 6 years ago

I second this request, and we discussed this idea with @saad-ali at this week's Storage SIG F2F. The chief concern is name collisions between storage class parameters and PVC annotations. Saad suggested this capability be gated via a startup flag. @orainxiong, are you planning to submit a PR for this?

orainxiong commented 6 years ago

@clintonk Thx for your replay. In order to solve this issue immediately, I handle this issue in a temporary way.

A new function, called appendPVCAnnotation, is introduced to encode PVC annotation to JSON string and modify options.parameter.

func appendPVCAnnotation(options controller.VolumeOptions) error {
    annotationPVC, err := json.Marshal(options.PVC.Annotations)
    if err != nil {
        return err
    }

    parameters := map[string]string{
        "annotationsPVC": string(annotation),
    }
    return nil
}

Note : Reverse key name annotationsPVC to avoid name conflict.

A new flag, called annotationPVCNeeded, is added to determine whether PVC annotation should be appended into options.parameter.

The advantage of implementation hardly modifies existing external-provisioner logic. Honestly, it is a not a preferable way.
Looking forward to your suggestions.

jsafrane commented 6 years ago

While I agree that there should be a way how to pass parameters from user to CreateVolume, I don't like suggested solution:

It makes the CSI driver Kubernetes aware. Whole point of CSI is to have independent plugins that can be used in Kubernetes, Mesos, Cloud Foundry, Docker Swarm and others. The plugins should not know anything about PVCs, these are pure Kubernetes objects.
Kubernetes admin needs a way how to restrict what parameters can be set by users. Kubernetes is (or can be) multi tenant system and not all users can use all parameters that a CSI plugin exposes.
User should be allowed to list possible parameters together with some description so they can choose the right value. Note that there may be GUI between users and Kubernetes, so the GUI must be able at least to enumerate these parameters. We tried in https://github.com/kubernetes/community/pull/543, however, nobody has stepped in to finish the proposal and implemented it.

fatih commented 6 years ago

After creating the csi-digitalocean driver, I also received a feedback, that needs this feature: https://github.com/digitalocean/csi-digitalocean/issues/30

I'm using the name of the request ot create the volumes, which ends up being the PV name. So it's in the form of kubernetes-dynamic-pv-<randomid>. However on the UI (at DigitalOcean.com), this is not useful because it doesn't give the person an easy way to track back to which entity it belongs to.

I thought maybe I can use the StorageClass parameters and add a new field called volumePrefix, such as:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: do-block-storage
  namespace: kube-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.digitalocean.csi.dobs
parameters:
  volumePrefix: "csi-digitalocean"

and then in the Controller plugin, I'll change the name in CreateVolume according to this information:

volumeName := req.Name
// change the name if the storage-class has a prefix option
prefix, ok := req.Parameters["volumePrefix"]
if ok {
    volumeName = prefix + "-" + volumeName
}

This allows me to create a volume based on this users CSI deployment. However, this is not scalable because:

you only create a single StorageClass, so it's not easy to update it every single time
you're not allows to update the parameters field of a StorageClass
StorageClass is global, whereas we need a local information from a local resource

A better idea would be to use the annotations of a PVC. This would allow us to create volume's directly based on the information bound to a specific PVC.

We're already passing down the information from a Kubernetes resource (i.e StorageClass) down to the plugin, I think there is not much that prevents us to pass the information fo another resource (i.e PersistentVolumeClaim) to the plugin.

sbezverk commented 6 years ago

check this out: https://github.com/kubernetes-csi/external-provisioner/blob/master/cmd/csi-provisioner/csi-provisioner.go#L46 I think you can leverage this parameter.

fatih commented 6 years ago

@sbezverk thanks for the tip, I didn't know it.

Though, the prefix was just an example, it could be also something else. Such tagging the resource on the cloud provider side based on the created PVC labels.

orainxiong commented 6 years ago

@jsafrane Thx for your reply.

If I understand correctly, I notice that external-provisioner has exposed StorageClass parameters to end storage drivers through VolumeOptions.

    options := VolumeOptions{
        PersistentVolumeReclaimPolicy: reclaimPolicy,
        PVName:     pvName,
        PVC:        claim,
        Parameters: parameters,  -- Store `StorageClass` parameters here
    }

    volume, err = ctrl.provisioner.Provision(options)

End storage drivers can get StorageClass parameters depending on their own logic. However, PVC's annotation is still missing in VolumeOptions.

I very agree with your opinion. It's a temporary solution which makes storage driver Kubernetes aware. It is a very time consuming job to figure out a COs neutral solution. As far as I think of parameters and description, it is very likely to be exhaust to create a common body of naming language across COs.

Looking forward to your suggestions.

jsafrane commented 6 years ago

@fatih, for tagging we could perhaps extend CSI's CreateVolume call with optional metadata such as Kubernetes PVC namespace + name, PV name, storage class name (?) or PVC labels (which one? all of them?). It would be used only to tag volumes in the storage backend so users can map these volumes to CO objects. These cannot be used to configure provisioned volume (see https://github.com/kubernetes-csi/external-provisioner/issues/86#issuecomment-390634816 above).

CO would provide generic map[string]string, opaque to CSI driver and CSI driver would blindly "attach" it to the volume via AWS tags or GCE labels or OpenStack metadata or whatever the storage supports (or throw them away if the storage does not support tagging).

However, PVC's annotation is still missing in VolumeOptions.

@orainxiong, that's intentional.

I could imagine our CSI provisioner passes content of some alpha annotations from PVC as CreateVolumeRequest.parameters in CreateVolume call, but these should be really alpha. To get the to GA, we need policy and discovery and whatnot.

fatih commented 6 years ago

for tagging we could perhaps extend CSI's CreateVolume call with optional metadata such as Kubernetes PVC namespace + name, PV name, storage class name (?) or PVC labels (which one? all of them?).

I think passing the PVC metadata would be great. Because it already includes the following important bits:

name
namespace
labels
annotations

I'm not sure how to pass them via CO though. a map[string]string is the easiest way. But then how are we going to represent nested maps? We could pass it as a JSON maybe:

map[string]string{
 "pvc" : "{\"name\": ...}",
 "storageclass": " { ... }",
}

I know that Kubernetes annotations mostly store data like this and it's a common thing there.

CO would provide generic map[string]string, opaque to CSI driver and CSI driver would blindly "attach" it to the volume via AWS tags or GCE labels or OpenStack metadata or whatever the storage supports (or throw them away if the storage does not support tagging).

Yeap. Attaching is only one of the things the CSI driver can accomplish. Changing the name is another action it should be allowed to do.

orainxiong commented 6 years ago

@jsafrane Thanks. I got it. I will close this issue.

fatih commented 6 years ago

Why is this closed? Did we have a PR that fixes this issue or a solution that is applicable right now?

orainxiong commented 6 years ago

@fatih

At least, in my opinion, there is a temporary way to handle this issue. But, it isn't applicable for all COs that support CSI.

As your suggestions, a new function injectCOParameter is introduced into external-provisioner :

func injectCOParameter(options controller.VolumeOptions) (map[string]string, error) {

    annotationsPVC, err := json.Marshal(options.PVC.Annotations)
    if err != nil {
        return nil, err
    }

    labelsPVC, err := json.Marshal(options.PVC.Labels)
    if err != nil {
        return nil, err
    }

    storageClass, err := json.Marshal(options.Parameters)
    if err != nil {
        return nil, err
    }

    parameters := map[string]string{
        "annotationsPVC": string(annotationsPVC),
        "labelsPVC":     string(labelsPVC),
        "storageClass":  string(storageClass),
    }
    return parameters, nil
}

injectCOParameter is in the charge of encoding labels and annotations and storageclass into CreateVolumeRequest.Parameters. Specifically, CreateVolumeRequest.Parameters is defined by CSI to store opaque parameter and COs will ignore it.

On the storage provisioner side, they can decode parameters from CreateVolumeRequest.Parameters according to their own operational logic.

fatih commented 6 years ago

@orainxiong Thanks for the reply. The title says Adding PVC's annotation to CreateVolumeRequest.Parameters and unless I'm missing something, there is no such feature and no one contributed to the provisioner/attacher to add this feature. Hence my question why we have closed this issue. Nothing is fixed yet and it's not possible to pass any PVC/PV data to the plugin from the CO. This issue should be re-opened, and closed when there is an action task we can work together.

orainxiong commented 6 years ago

@fatih ok, you're right. : - )

mkimuram commented 6 years ago

@orainxiong

Has injectCOParameter() already been merged to external-provisioner? I couldn't find it in the codes nor PRs, so I would appreciate it if you could share any links to it.

orainxiong commented 6 years ago

@mkimuram Hope it works for you. https://github.com/orainxiong/external-provisioner/tree/issue-86

mkimuram commented 6 years ago

@orainxiong Thank you for providing code. Great work!

I tested your code as below and confirmed that annotationsPVC and labelsPVC were added to parameters that are passed to each driver. The concept looks good to me and will solve my concern(https://github.com/kubernetes-csi/external-provisioner/issues/93).

In my opinion, there would be still rooms for discussion to make this code merged:

What parameters will be needed to be passed, (I found that annotation for storage class is not passed, and there might be some other fields that will be useful if we pass.)
How they will be passed, (Should parameters for storageClass be passed directory to keep compatibility, and so on?)
Should they be passed in the first place, (Although, I personally believe that they should, it doesn't seem to be agreed in the community.)

I found a related discussion below for 3, and this will be needed to be discussed, first . https://github.com/container-storage-interface/spec/issues/248

Any comments for this issue? @jsafrane @princerachit @vladimirvivien Especially, I would like to know whether k8s can add this code as a temporary workaround, before we reach to a conclusion for above CSI discussion.

Summary of test Results: [Without patch]

# kubectl logs csi-pod hostpath-driver 
I0612 18:57:50.339548       1 utils.go:97] GRPC request: name:"pvc-80ee79686e7211e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"foo21" value:"bar21" > parameters:<key:"foo22" value:"bar22" >

[With patch]

# kubectl logs csi-pod hostpath-driver 
I0612 20:37:56.738211       1 utils.go:97] GRPC request: name:"pvc-7d0658ed6e8011e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"annotationsPVC" value:"{\"foo31\":\"bar31\",\"foo32\":\"bar32\",\"volume.beta.kubernetes.io/storage-provisioner\":\"csi-hostpath\"}" > parameters:<key:"labelsPVC" value:"{\"foo41\":\"bar41\",\"foo42\":\"bar42\"}" > parameters:<key:"storageClass" value:"{\"foo21\":\"bar21\",\"foo22\":\"bar22\"}" >

Please see below for how I tested:

[Without patch] (Also see [files] below for csi-sc2.yaml and csi-pvc2.yaml.) ``` # FEATURE_GATES=CSIPersistentVolume=true ALLOW_PRIVILEGED=true ALLOW_SECURITY_CONTEXT=true hack/local-up-cluster.sh # export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig # kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/docs/master/book/src/example/csi-setup.yaml # kubectl create -f csi-sc2.yaml # kubectl create -f csi-pvc2.yaml # kubectl logs csi-pod hostpath-driver | grep -A 3 "CreateVolume" | tail -4 I0612 18:57:50.339545 1 utils.go:96] GRPC call: /csi.v0.Controller/CreateVolume I0612 18:57:50.339548 1 utils.go:97] GRPC request: name:"pvc-80ee79686e7211e8" capacity_range: volume_capabilities: access_mode: > parameters: parameters: I0612 18:57:50.339719 1 controllerserver.go:87] create volume /tmp/80f0db83-6e72-11e8-b5be-0242ac110003 I0612 18:57:50.339724 1 utils.go:102] GRPC response: volume: attributes: > ``` [With patch] (Also see [files] below for csi-sc2.yaml and csi-pvc2.yaml.) ``` # git clone https://github.com/orainxiong/external-provisioner.git # cd external-provisioner # git checkout issue-86 # sed -i "s/canary/hack/" Makefile # make container # curl https://raw.githubusercontent.com/kubernetes-csi/docs/master/book/src/example/csi-setup.yaml | sed "s/quay.io\/k8scsi\/csi-provisioner:v0.2.1/quay.io\/k8scsi\/csi-provisioner:hack/" > csi-setup2.yaml # sed -i "s/imagePullPolicy: Always/imagePullPolicy: Never/" csi-setup2.yaml # FEATURE_GATES=CSIPersistentVolume=true ALLOW_PRIVILEGED=true ALLOW_SECURITY_CONTEXT=true hack/local-up-cluster.sh # export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig # kubectl create -f csi-setup2.yaml # kubectl create -f csi-sc2.yaml # kubectl create -f csi-pvc2.yaml # kubectl logs csi-pod hostpath-driver | grep -A 3 "CreateVolume" | tail -4 I0612 20:37:56.738207 1 utils.go:96] GRPC call: /csi.v0.Controller/CreateVolume I0612 20:37:56.738211 1 utils.go:97] GRPC request: name:"pvc-7d0658ed6e8011e8" capacity_range: volume_capabilities: access_mode: > parameters: parameters: parameters: I0612 20:37:56.738409 1 controllerserver.go:87] create volume /tmp/7d088fc3-6e80-11e8-84a2-0242ac110002 I0612 20:37:56.738416 1 utils.go:102] GRPC response: volume: attributes: attributes: > # kubectl logs csi-pod external-provisioner | grep -A 3 "CreateVolume" | tail -4 I0612 20:37:56.738068 1 controller.go:104] GRPC call: /csi.v0.Controller/CreateVolume I0612 20:37:56.738072 1 controller.go:105] GRPC request: name:"pvc-7d0658ed6e8011e8" capacity_range: volume_capabilities: access_mode: > parameters: parameters: parameters: I0612 20:37:56.738515 1 controller.go:107] GRPC response: volume: attributes: attributes: > I0612 20:37:56.738538 1 controller.go:108] GRPC error: ``` [Files] ``` # cat csi-sc2.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-hostpath-sc2 annotations: foo11: bar11 foo12: bar12 provisioner: csi-hostpath reclaimPolicy: Delete volumeBindingMode: Immediate parameters: foo21: bar21 foo22: bar22 ``` ``` # cat csi-pvc2.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-pvc2 annotations: foo31: bar31 foo32: bar32 labels: foo41: bar41 foo42: bar42 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: csi-hostpath-sc2 ```

princerachit commented 6 years ago

@mkimuram today's discussion concluded that we are not going to immediately touch the spec for CreateVolumeRequest to pass CO specific metadata(PVC name,PVC id etc). A proposed workaround is to write your own external provisioner which knows what params to pass for your specific driver.

However, as pvc annotations are also used to add volume related information, adding them to external-provisioner would be possible. We need someone from k8s wg-csi to nod on this. @saad-ali

orainxiong commented 6 years ago

@clintonk You're welcome. : - )

kfox1111 commented 5 years ago

We've talked about this a bit in https://github.com/kubernetes/community/pull/2273 as well.

I'd like a way to have VolumeAttributes be passed in PVC's and the external provisioner can decide if its ok merging none/some/all of them with the StorageClasses VolumeAttributes when sending to the CSI drivers Create/DeleteVolume. And also if none/some/all should be copied to the resulting PV for the CSI driver's mount/unmount.

This wouldn't be specific to Kubernetes, as they would be CSI attributes, not Kubernetes annotations.

mkimuram commented 5 years ago

@kfox1111

I'd like a way to have VolumeAttributes be passed in PVC's and the external provisioner can decide if its ok merging none/some/all of them with the StorageClasses VolumeAttributes when sending to the CSI drivers Create/DeleteVolume.

Is it the same logic to @orainxiong 's above code or different one?

According to @princerachit 's below comment, at least pvc annotations could be allowed to be passed by above way, if k8s wg-csi agrees.

However, as pvc annotations are also used to add volume related information, adding them to external-provisioner would be possible. We need someone from k8s wg-csi to nod on this. @saad-ali

saad-ali commented 5 years ago

My current standing on this issue: Annotations on PVCs MUST NOT be passed to CSI drivers. The Kubernetes PVC object is intended for application portability. When we start leaking cluster/implementation specific details in to it we are violating that principle. And explicitly passing PVC annotations to CSI drivers encourages that pattern. Workload portability is a corner stone of Kubernetes. It is one of the major reasons for the success of the project. We must be very careful when we violate it.

Instead let's focus on specific use cases. And let's see if we can come up with better solutions for each of those uses cases rather then opening up a hole in the API.

mlmhl commented 5 years ago

@saad-ali There is a similar use case in my team:

We support multiple storage types in a cluster, such as Ceph RBD and CephFS. What's more, we also support multiple file system types for Ceph RBD, such as ext4, xfs, etc. The only solution at present is creating a StorageClass for each file system type, for example, a ext4-sc StorageClass for ext4 fs type, and a xfs-sc StorageClass for xfs fs type.

However, we also need to set quota for PVCs, so we need to set quota for these two StorageClasses(ext4-sc and xfs-sc) separately if we adopt this approach. This is strange for users because these two StorageClasses are the same storage type(Ceph RBD), just different file systems.

jsafrane commented 5 years ago

StorageClass object was supposed to provide classes of storage ("slow", "fast", "secure"), not configuration of storage ("ssd-4-replicas-xfs", "ssd-4-replicas-ext4", "ssd-3-replicas-xfs"...)

Why should user need to choose between say ext2 or xfs? Users need storage to run say "fast cache", where ext2 simplicity and speed is better than xfs or "database XYZ", where xfs is more stable than ext2 and XYZ is more optimized for xfs than ext4 (all examples are completely fictional).

So there should be a class "fast cache", "database XYZ" and "default". How many such classes do you need in a cluster? Maybe lot of them, but their number grows with number of applications and their different needs, not with number of configuration options each storage provides.

j-griffith commented 5 years ago

On Tue, Mar 19, 2019 at 10:01 AM Jan Šafránek notifications@github.com wrote:

StorageClass object was supposed to provide classes of storage ("slow", "fast", "secure"), not configuration of storage ("ssd-4-replicas-xfs", "ssd-4-replicas-ext4", "ssd-3-replicas-xfs"...)

Why should user need to choose between say ext2 or xfs? Users need storage to run say "fast cache", where ext2 simplicity and speed is better than xfs or "database XYZ", where xfs is more stable than ext2 and XYZ is more optimized for xfs than ext4 (all examples are completely fictional).

So there should be a class "fast cache", "database XYZ" and "default". How many such classes do you need in a cluster? Maybe lot of them, but their number grows with number of applications and their different needs, not with number of configuration options each storage provides.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kubernetes-csi/external-provisioner/issues/86#issuecomment-474443188, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLKjKdUsWXko_NLVlsPec45p-q8r4Tmks5vYQnVgaJpZM4T9H5P .

I agree with NOT passing down the attributes, if we really need something more than what's offered by the SC model then I think we would need to consider a new mechanism (like special parameters per volume instead of class [note I'm not suggesting that we discuss that here or go that route necessarily]).

One thing @mimhl that I think might be a more elegant way to solve your problem with the FS types for example; you could use block mode, and then put whatever file-system you want on the volume, either as the user or perhaps via a side car pattern or some other worker pod to initialize it, the block model gives you the flexibility to do a lot of different things that you're currently relying on storage classes for.

ayanamist commented 5 years ago

We have similar situation. @jsafrane The word "user" today means too many things. Do you mean "the end user" which equals to DevOps or SRE, or "PaaS user" which build their ability on the k8s? We are the latter one who focus on internal operations of a company, and yes the end user do not care about ext4 or xfs, but we care. We do not expose the whole k8s abstraction to the end user. So we need detailed customization rather than several wild classes.

miaowing commented 5 years ago

How portworx openstorage get pvc annotations in csi driver?

pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc
  annotations:
    openstorage.io/auth-secret-name: example-secret-name
    openstorage.io/auth-secret-namespace: example-secret-namespace
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: storage-class

csi controller server:

func (s *OsdCsiServer) CreateVolume(
    ctx context.Context,
    req *csi.CreateVolumeRequest,
) (*csi.CreateVolumeResponse, error) {
        // Get parameters
    spec, locator, source, err := s.specHandler.SpecFromOpts(req.GetParameters())
    if err != nil {
        e := fmt.Sprintf("Unable to get parameters: %s\n", err.Error())
        logrus.Errorln(e)
        return nil, status.Error(codes.InvalidArgument, e)
    }
}

openstorage server:

spec := dcReq.GetSpec()
locator := dcReq.GetLocator()
secretName, secretContext, err := a.parseSecret(spec.VolumeLabels, locator.VolumeLabels, true)

pohly commented 5 years ago

How portworx openstorage get pvc annotations in csi driver?

I think they use a fork of external-provisioner for this. @lpabon might be able to confirm that.

@kerneltime mentioned another external-provisioner fork here: https://github.com/kubernetes-csi/external-provisioner/issues/170#issuecomment-485008849

My take on this: when multiple people come to the conclusion that they need to fork external-provisioner to implement their use cases, then this is an indication that something is missing in what is provided upstream. We might want to reconsider that...

taherv commented 4 years ago

StorageClass object was supposed to provide classes of storage ("slow", "fast", "secure"), not configuration of storage ("ssd-4-replicas-xfs", "ssd-4-replicas-ext4", "ssd-3-replicas-xfs"...)

Why should user need to choose between say ext2 or xfs? Users need storage to run say "fast cache", where ext2 simplicity and speed is better than xfs or "database XYZ", where xfs is more stable than ext2 and XYZ is more optimized for xfs than ext4 (all examples are completely fictional).

So there should be a class "fast cache", "database XYZ" and "default". How many such classes do you need in a cluster? Maybe lot of them, but their number grows with number of applications and their different needs, not with number of configuration options each storage provides.

@jsafrane This argument does not work for the enterprise. The problem of how to use a filesystem efficiently for any given application is complex, and discerning enterprise admins know more about their application than any orchestrator/storage vendor can. Not only is the choice of filesystem important, but to make matters worse, each filesystem has tradeoffs between integrity and performance. We have seen customers solve this problem on a per application basis by setting custom mount options. For e.g ordered-data mode (journaling mode) provides better integrity of the ext-x family of filesystems, but of course it is slower. Some db vendors recommend integrity settings for their log volumes, but performance-biased settings for their data volumes.

Long story short, I think there is a need for per-volume, volume-type-specific parameters/configuration that enterprise users will want. I agree storage classes are not the right tool to use it. I also agree with @saad-ali that annotations break abstractions.

If the community agrees about NOT supporting annotations, that decision should be accompanied by an alternative approach. Otherwise, the CSI API falls short of expectations.

My suggestion : A sidecar release that supports annotations as "deprecated parameters" could serve as a transition path. That will prevent forks of sidecars (which is a big win for the end user). The problem then becomes the discussion of which use-cases should be promoted to first class CSI API ..... and also how to end-of-life this sidecar.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

pohly commented 4 years ago

/remove-lifecycle stale

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

sermilrod commented 4 years ago

/remove-lifecycle stale

sermilrod commented 4 years ago

My current standing on this issue: Annotations on PVCs MUST NOT be passed to CSI drivers. The Kubernetes PVC object is intended for application portability. When we start leaking cluster/implementation specific details in to it we are violating that principle. And explicitly passing PVC annotations to CSI drivers encourages that pattern. Workload portability is a corner stone of Kubernetes. It is one of the major reasons for the success of the project. We must be very careful when we violate it.

Instead let's focus on specific use cases. And let's see if we can come up with better solutions for each of those uses cases rather then opening up a hole in the API.

Our use case at Ticketmaster is tagging EBS volumes when we create PVCs. We use tags to associate costs and we would need a way to pass additional tags to PVCs. Our current workaround is a controller that watches PVCs and based on an annotation it tags the volumes.

I do not see any difference between this use case and the use case of creating a load balancer and adding an annotation to tag the resource:

service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "Tag1=value1,Tag2=value2"

This is creating a resource in AWS, the same the CSI driver does when creating a PVC, and in the case of the LoadBalancer there is an annotation to handle tags. Why can't we have the same user experience?

Nilesh20 commented 4 years ago

@saad-ali Our use case is to add the details pf PVC (pvc name, Uuid) as metadata to resource created on storage array. is there anyway to achieve this as part of create request. Initially in external storage we were able to grab these details.

yoesmat commented 4 years ago

FYI: PR: add pvc metadata to createvolume req https://github.com/kubernetes-csi/external-provisioner/pull/399

zetsub0u commented 4 years ago

@Nilesh20 i need to do the same thing, add metadata in the storage arrays, the approach in #399 is to have the minimum information needed to then go to the kubernetes api to get anything else we need from the plugin, so in this case having the pvc namespace/name is enough to find whatever we need.

gregnsk commented 4 years ago

We would like to allow customers to "import" an existing volume as a PVC, something similar to OpenStack cinder manage functionality. Think about a volume created as a replica from another storage array, or restored from some external backup device, or created with older dynamic volume provisioner - outside of the CSI driver.

@saad-ali - If the annotations are not supported, what would be the best way to transfer the actual volume name to the CSI driver?

jsafrane commented 4 years ago

@gregnsk, there is https://github.com/kubernetes-csi/external-provisioner/pull/399, where a driver can optionally get PV/PVC name. My use case is to "tag" / "label" the volume so if Kubernetes dies from whatever reason, storage admin still can find which volume is what.

But the driver can do anything with this information and we can't prevent it reading PVC annotations from Kubernetes API server. *cough* *cough*

msau42 commented 4 years ago

@gregnsk for your use case, you can manually create a PV object to import a pre-existing volume

gregnsk commented 4 years ago

The problem is that I want to allow management of this imported PV via CSI, so it can't remain static allocation

On Wed, Mar 4, 2020 at 7:35 AM Michelle Au notifications@github.com wrote:

@gregnsk https://github.com/gregnsk for your use case, you can manually create a PV object to import a pre-existing volume

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-csi/external-provisioner/issues/86?email_source=notifications&email_token=ADDNUPPU44VP4EATTKHIG73RFZYMBA5CNFSM4E7UPZH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENYPTHY#issuecomment-594606495, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDNUPMDWP54FOHR5U36Q7LRFZYMBANCNFSM4E7UPZHQ .

msau42 commented 4 years ago

What kind of management are you looking for? Once you manually create a PV and bind it to a PVC, you can use it like any other dynamically provisioned volume

gregnsk commented 4 years ago

expand/clone/snapshot/delete. All CSI functionality.

On Wed, Mar 4, 2020 at 7:47 AM Michelle Au notifications@github.com wrote:

What kind of management are you looking for? Once you manually create a PV and bind it to a PVC, you can use it like any other dynamically provisioned volume

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-csi/external-provisioner/issues/86?email_source=notifications&email_token=ADDNUPJLXWWR4AWSPWJMK2LRFZZYRA5CNFSM4E7UPZH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENYR4MQ#issuecomment-594615858, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDNUPKSEVCKLEVAYEKQ7L3RFZZYRANCNFSM4E7UPZHQ .

msau42 commented 4 years ago

expand/clone/snapshot/delete. All CSI functionality.

All these operations can be done for manually created PVs as long as you associate them with a storageclass

gregnsk commented 4 years ago

interesting. how can I associate a manually created PV with a storage class?

On Wed, Mar 4, 2020 at 8:39 AM Michelle Au notifications@github.com wrote:

expand/clone/snapshot/delete. All CSI functionality.

All these operations can be done for manually created PVs as long as you associate them with a storageclass

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-csi/external-provisioner/issues/86?email_source=notifications&email_token=ADDNUPOOVW75RAJLACT7FZTRFZ73PA5CNFSM4E7UPZH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENY2HHY#issuecomment-594650015, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDNUPON6J5QU25NVQYCKITRFZ73PANCNFSM4E7UPZHQ .

msau42 commented 4 years ago

interesting. how can I associate a manually created PV with a storage class?

PV.spec.storageclassname. And also set the same storageclassname on the PVC.

haibinxie commented 4 years ago

We also need to import an existing volume with CSI, what's the timeline for next release of external provisioner to take PVC name in CSI (#399 )

msau42 commented 4 years ago

@haibinxie you don't need #399 to import an existing volume with CSI. This is already supported through manually creating a PV.

msau42 commented 4 years ago

@haibinxie @gregnsk here's an example driver documentation on how to manually create a PV for an existing disk: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/docs/kubernetes/user-guides/pre-provisioned.md

haibinxie commented 4 years ago

@msau42 Thanks for sharing the info, in that example how do you guarantee the PV would not be bound to other PVC before you create PVC, i have seen others set claimRef in PV to ensure that.

kubernetes-csi / external-provisioner

Adding PVC's annotation to CreateVolumeRequest.Parameters #86