dpaks commented 3 years ago

Describe the bug

I'm trying to unsuccessfully create a k8s pod with inline CSI filesystem static volume. The static volume mount works if I create a persistent volume.

Environment details

Image/version of Ceph CSI driver : ceph/ceph:v15.2.4
Helm chart version : rook's v1.4.2
Kernel version : 5.0.0-1029-gcp
Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its krbd or rbd-nbd) : No PVC involved
Kubernetes cluster version : 1.15.3
Ceph cluster version : v15.2.4

Steps to reproduce

Apply foll. yaml

apiVersion: v1
kind: Pod
metadata:
  name: inline-csicephfs-pod
  namespace: xx-infra
spec:
  containers:
  - name: eman
    image: ubuntu:20.04
    command: ["sleep", "infinity"]
    volumeMounts:
    - name: vol
      mountPath: /mnt
  volumes:
  - name: vol
    csi:
      driver: xx-infra.cephfs.csi.ceph.com
      nodePublishSecretRef:
        name: csi-cephfs-secret
      volumeAttributes:
        "clusterID": "xx-infra"
        "fsName": "myfs"
        "staticVolume": "true"
        "rootPath": "/"

Actual results

The Pod "inline-csicephfs-pod" is invalid:

spec.volumes[0]: Required value: must specify a volume type
spec.containers[0].volumeMounts[0].name: Not found: "vol"

Expected behavior

Successful pod creation

Additional context

If I create the foll. PV, I can mount it in the pod.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: myPV
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: ""
  csi:
    driver: xx-infra.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: csi-cephfs-secret
      namespace: xx-infra
    volumeAttributes:
      "clusterID": "xxx-infra"
      "fsName": "myfs"
      "staticVolume": "true"
      "rootPath": "/"
    volumeHandle: rand
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

dpaks commented 3 years ago

https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html Will it work if I add --feature-gates=CSIInlineVolume=true since I use k8s 1.15?

humblec commented 3 years ago

@dpaks havent experimented this, let me give a try. I will get back soon.

humblec commented 3 years ago

one general thought here is, the inline volume got better support with kube 1.19. @dpaks , if at all you have this version of cluster , can you give a try ?

dpaks commented 3 years ago

I enabled feature gate on 1.15 and pod creation API succeeded. However, mount failed. I didn't yet check the logs though. Warning FailedMount 36s kubelet, dpak-m1 Unable to mount volumes for pod "inline-csicephfs-pod_xx-infra(77116229-3c3a-4754-9de0-35bacc79abd7)": timeout expired waiting for volumes to attach or mount for pod "xx-infra"/"inline-csicephfs-pod". list of unmounted volumes=[vol]. list of unattached volumes=[vol default-token-mxwtv]

@humblec sorry, I don't have a 1.19 cluster.

humblec commented 3 years ago

@dpaks any luck if you specify nodestagesecretref too ?

dpaks commented 3 years ago

Pod spec uses https://godoc.org/k8s.io/api/core/v1#CSIVolumeSource which sadly doesn't have nodestagesecretref.

humblec commented 3 years ago

Pod spec uses https://godoc.org/k8s.io/api/core/v1#CSIVolumeSource which sadly doesn't have nodestagesecretref.

I see, let me spend some time on this. Will get back to you.

Madhu-1 commented 3 years ago

@humblec as this is assigned to you please add an E2E and documentation for the same.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

mykaul commented 3 years ago

@humblec - was anything done here?

Madhu-1 commented 3 years ago

it's possible to support this as a new feature. reopening for the same reason.

Madhu-1 commented 2 years ago

@humblec if you are not working on this one anymore can you please unassign it so someone else can take it up?

humblec commented 2 years ago

@humblec if you are not working on this one anymore can you please unassign it so someone else can take it up?

Have done some experiments here @Madhu-1 , so let me revisit the same. If help is requred, I will reassign

Madhu-1 commented 2 years ago

@humblec if you are not working on this one anymore can you please unassign it so someone else can take it up?

Have done some experiments here @Madhu-1 , so let me revisit the same. If help is requred, I will reassign

yes please, this is pending for a long time. planning to work on this one if there is no draft PR/design yet.

humblec commented 2 years ago

@humblec if you are not working on this one anymore can you please unassign it so someone else can take it up?

Have done some experiments here @Madhu-1 , so let me revisit the same. If help is requred, I will reassign

yes please, this is pending for a long time. planning to work on this one if there is no draft PR/design yet.

A couple of things to note here, the inline ephemeral volume support has been undergoing bit heavy changes in upstream in last kube releases and different modes were introduced in last few kube releases to support this functionality for different scenarios. This feature support is still not GAd. That said, inline ephemeral support is still in BETA state upstream kubernetes.

Regardless, these features were getting explored to a good extent in between. I have captured whatever I had in my notes here https://hackmd.io/SXcB4VizTKaI4E7efOV_xw and filed https://github.com/ceph/ceph-csi/issues/2587 with the design doc . Please feel free to take a look at it and comment if any. Once we are good on the design, implementation PR will be filed. Thanks.

Madhu-1 commented 2 years ago

@humblec if you are not working on this one anymore can you please unassign it so someone else can take it up?

Have done some experiments here @Madhu-1 , so let me revisit the same. If help is requred, I will reassign

yes please, this is pending for a long time. planning to work on this one if there is no draft PR/design yet.

A couple of things to note here, the inline ephemeral volume support has been undergoing bit heavy changes in upstream in last kube releases and different modes were introduced in last few kube releases to support this functionality for different scenarios. This feature support is still not GAd. That said, inline ephemeral support is still in BETA state upstream kubernetes.

Regardless, these features were getting explored to a good extent in between. I have captured whatever I had in my notes here https://hackmd.io/SXcB4VizTKaI4E7efOV_xw and filed #2587 with the design doc . Please feel free to take a look at it and comment if any. Once we are good on the design, implementation PR will be filed. Thanks.

@humblec Thanks for the design doc. IMO the design doc contains the same or most of the information from https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html does not contain more required implementation details which are required in cephcsi nor the major challenges.

the doc only talks about rbd nothing related to cephfs as the requirement is for the ceph-csi drivers.

I see 3 major problems or missing pieces in the design

Looks like you have not considered the missing secret in the NodeUnstage which is required for cleanup
Generation/deriving the rbd image/cephfs subvolume name
Storing of the volume/mount-related pieces of information which are required for delete/unmap operations.

humblec commented 2 years ago

the doc only talks about rbd nothing related to cephfs as the requirement is for the ceph-csi drivers.

I see 3 major problems or missing pieces in the design
* Looks like you have not considered the missing secret in the NodeUnstage which is required for cleanup

* Generation/deriving  the rbd image/cephfs subvolume name

* Storing of the volume/mount-related pieces of information which are required for delete/unmap operations.

These are bit more on the implementation side, not sure it has to be mentioned in the design doc. But I can add those as well in the same if required. As mentioned earlier, please feel free to comment on the design doc in line of any other information you would like to see in the doc, so that I can fill it and continue.

Madhu-1 commented 2 years ago

the doc only talks about rbd nothing related to cephfs as the requirement is for the ceph-csi drivers. I see 3 major problems or missing pieces in the design
* Looks like you have not considered the missing secret in the NodeUnstage which is required for cleanup

* Generation/deriving  the rbd image/cephfs subvolume name

* Storing of the volume/mount-related pieces of information which are required for delete/unmap operations.
These are bit more on the implementation side, not sure it has to be mentioned in the design doc. But I can add those as well in the same if required. As mentioned earlier, please feel free to comment on the design doc in line of any other information you would like to see in the doc, so that I can fill it and continue.

IMHO this is a design enhancement/change I would like to see these things in the design doc without these high-level details it's difficult to understand the flow/design to review or to do suggestions. and it's good to open a GitHub PR for the design doc instead of using hackmd for the design and suggestions.

Madhu-1 commented 8 months ago

Am closing this one as this cannot be done in cephcsi due to the existing limitations discussed above, please reopen if required.

ceph / ceph-csi

Inline Volume Support for K8s Pod #1461

Describe the bug

Environment details

Steps to reproduce

Actual results

Expected behavior

Additional context