volume mode "Persistent" not supported by driver dobs.csi.digitalocean.com

[AATHITH]

What did you do?

I tried installing mysql with pv, mysql pod stuck with the following error:

MountVolume.NewMounter initialization failed for volume "pvc-4765c451-c31c-4090-807c-66c251247126" : volume mode "Persistent" not supported by driver dobs.csi.digitalocean.com (only supports [])

what does this mean? what should I do now??

What did you expect to happen?

I expect redis-ha to be attached to PV and work fine.

Configuration:

• CSI Version: 1.2.0

• Kubernetes Version: 1.16.0

• Cloud provider: Digital Ocean

[timoreimann]

Hi there

Thanks for reporting. Interesting, we should be supporting Persistent volumes for sure. Let me spend some time to dig into this and see what the matter is.

[timoreimann]

Quick question: how did you provision your cluster? kubeadm, kops, something else?

[timoreimann]

Diving into the Kubernetes code, it looks like this is related to the capabilities announced by the CSIDriver object. Kubernetes 1.16 introduced the volumeLifecycleModes field which can have a value of Persistent. We do not set that in our release files, but we shouldn't have to because it's the default according to the documentation. On a 1.16 test cluster of mine using our CSIDriver manifest, I see the Persistent value set. (Kubernetes probably adds it by default if it's missing.)

@AATHITH could you share the output of kubectl get csidriver dobs.csi.digitalocean.com -o yaml please? Thank you.

[AATHITH]

Quick question: how did you provision your cluster? kubeadm, kops, something else?

I created my cluster with kubeadm

[AATHITH]

Diving into the Kubernetes code, it looks like this is related to the capabilities announced by the CSIDriver object. Kubernetes 1.16 introduced the volumeLifecycleModes field which can have a value of Persistent. We do not set that in our release files, but we shouldn't have to because it's the default according to the documentation. On a 1.16 test cluster of mine using our CSIDriver manifest, I see the Persistent value set. (Kubernetes probably adds it by default if it's missing.)

@AATHITH could you share the output of kubectl get csidriver dobs.csi.digitalocean.com -o yaml please? Thank you.

i have downgraded to k8 1.15, its working without issues.

[timoreimann]

@AATHITH would you mind sharing the output of that kubectl command on a 1.16 cluster anyway? The reason it's possibly working on 1.15 is that the related volumeLifecycleModes field was only added in 1.16.

Thanks!

[AATHITH]

kubectl get csidriver dobs.csi.digitalocean.com -o yaml

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"CSIDriver","metadata":{"annotations":{},"name":"dobs.csi.digitalocean.com"},"spec":{"attachRequired":true,"podInfoOnMount":true}}
  creationTimestamp: "2020-01-20T03:56:35Z"
  name: dobs.csi.digitalocean.com
  resourceVersion: "4308503"
  selfLink: /apis/storage.k8s.io/v1beta1/csidrivers/dobs.csi.digitalocean.com
  uid: 130defc3-9ba4-46ac-8742-a7040307eabc
spec:
  attachRequired: true
  podInfoOnMount: true

[timoreimann]

@AATHITH ah interesting, the field is missing for you. This is what the object looks like on my DOKS 1.16.2-do.2 cluster:

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"CSIDriver","metadata":{"annotations":{},"name":"dobs.csi.digitalocean.com"},"spec":{"attachRequired":true,"podInfoOnMount":true}}
  creationTimestamp: "2019-11-21T10:06:51Z"
  name: dobs.csi.digitalocean.com
  resourceVersion: "197"
  selfLink: /apis/storage.k8s.io/v1beta1/csidrivers/dobs.csi.digitalocean.com
  uid: 52eb1d4f-59b5-46fc-8340-03b77eb1c9cd
spec:
  attachRequired: true
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent

I bet that if you add that Persistent list item, it's going to work for you. Perhaps you can try that?

I am going to do some 🕵-ing on why volumeLifecycleModes looks different to you. A bug on 1.16.0 came to my mind first, though I cannot find anything relevant in the Kubernetes change log.

[timoreimann]

@AATHITH I was able to determine one behavior that may have an influence in this case: if the CSIInlineVolume feature flag is disabled, then the Persistent value would not be included in volumeLifecycleModes by default.

In Kubernetes 1.16, that feature flag is on by default; so in order for this to make a difference, it'd need to be disabled explicitly. Can you confirm if this was or was not the case?

[pohly]

If it is disabled, then MountVolume.NewMounter should also be disabled. That part may be faulty in Kubernetes.

[timoreimann]

Closing for now since we haven't heard from the OP anymore. If this is still an issue, feel free to ping again and I'll reopen.