volumeNamePrefix parameter is ignored when registering rbd and cephfs volumes

[crltc]

Describe the bug

The volumeNamePrefix parameter is ignored when registering rbd and cephfs volumes. Note that this only occurs when registering existing volumes that have a prefix and works correctly when creating a new volume through ceph-csi.

Environment details

• Image/version of Ceph CSI driver: quay.io/cephcsi/cephcsi:v3.10.2
• Nomad version: 1.7.5
• Kernel version : 6.1.0-18-amd64
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its krbd or rbd-nbd) : rbd and kernel
• Ceph cluster version : 18.2.1 reef (stable)

Steps to reproduce

Steps to reproduce the behavior:

Setup Details: I have a near identical setup to the example provided here: https://docs.ceph.com/en/latest/rbd/rbd-nomad/

Deployment: Create cephfs volume in cephfs with the name below:

csi-vol-example-084f01c4-9a21-4b5a-a23d-5045e47cb6fz

Run nomad volume job with Terraform:

resource "nomad_csi_volume_registration" "example" {
  external_id = "0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz"
  volume_id = "ceph-example"
  name = "ceph-example"
  plugin_id = "cephfs-csi"
  context = {
    clusterID = "090e7e3f-0b34-289q-9202-gh8z35g3db51"
    fsName = "examplefs"
    volumeNamePrefix = "csi-vol-example-"
  }
  secrets = {
    adminID  = "redacted"
    adminKey = "redacted"
  }
  capability {
    access_mode     = "multi-node-multi-writer"
    attachment_mode = "file-system"
  }
  mount_options {
    fs_type = "ext4"
  }
}

Actual results

Volume mount and registration fail

Expected behavior

The volume should mount and register successfully. When I create a different cephfs volume named csi-vol-084f01c4-9a21-4b5a-a23d-5045e47cb6fz without a non-standard volume prefix - it mounts and registers successfully.

Logs

0320 02:11:04.698300       1 utils.go:164] ID: 25370 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC call: /csi.v1.Node/NodeStageVolume
I0320 02:11:04.698409       1 utils.go:165] ID: 25370 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC request: {"secrets":"***stripped***","staging_target_path":"/local/csi/staging/ceph-example/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"090e7e3f-0b34-289q-9202-gh8z35g3db51","fsName":"examplefs","volumeNamePrefix":"csi-vol-example-"},"volume_id":"0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz"}
E0320 02:11:04.701327       1 omap.go:79] ID: 25370 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz omap not found (pool="examplefs_metadata", namespace="csi", name="csi.volume.084f01c4-9a21-4b5a-a23d-5045e47cb6fz"): rados: ret=-2, No such file or directory
W0320 02:11:04.701353       1 voljournal.go:729] ID: 25370 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0320 02:11:04.703415       1 volume.go:163] ID: 25370 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz failed to get subvolume info for the vol csi-vol-example-084f01c4-9a21-4b5a-a23d-5045e47cb6fz: rados: ret=-2, No such file or directory: "subvolume 'csi-vol-example-084f01c4-9a21-4b5a-a23d-5045e47cb6fz' does not exist"
E0320 02:11:04.703480       1 utils.go:169] ID: 25370 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = volume not found
I0320 02:11:05.054617       1 utils.go:164] ID: 25371 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0320 02:11:05.054661       1 utils.go:165] ID: 25371 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC request: {"target_path":"/local/csi/per-alloc/68540888-9bd5-ad1e-8a41-d7ebed319477/ceph-example/rw-file-system-single-node-writer","volume_id":"0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz"}
E0320 02:11:05.054693       1 nodeserver.go:551] ID: 25371 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz stat failed: stat /local/csi/per-alloc/68540888-9bd5-ad1e-8a41-d7ebed319477/ceph-example/rw-file-system-single-node-writer: no such file or directory
I0320 02:11:05.054705       1 nodeserver.go:555] ID: 25371 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz targetPath: /local/csi/per-alloc/68540888-9bd5-ad1e-8a41-d7ebed319477/ceph-example/rw-file-system-single-node-writer has already been deleted
I0320 02:11:05.054714       1 utils.go:171] ID: 25371 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC response: {}
I0320 02:11:05.055046       1 utils.go:164] ID: 25372 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC call: /csi.v1.Node/NodeUnstageVolume
I0320 02:11:05.055093       1 utils.go:165] ID: 25372 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC request: {"staging_target_path":"/local/csi/staging/ceph-example/rw-file-system-single-node-writer","volume_id":"0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz"}
I0320 02:11:05.055133       1 utils.go:171] ID: 25372 Req-ID: 0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz GRPC response: {}

[Madhu-1]

@crltc yes its not expected to set the volumeNamePrefix for pre created volumes because it will be ignored, i don't have example for nomad but this is how the kubernetes static PVC's looks like https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md#cephfs-static-pvc

[crltc]

Thank you, with that info I was able to successfully provision the existing volume on Nomad. Included my config below for anyone else who may come across this. I also had to change adminID to userID (and still using admin credentials, not sure if required).

resource "nomad_csi_volume_registration" "example" {
  external_id = "/volumes/csi/csi-vol-example-084f01c4-9a21-4b5a-a23d-5045e47cb6fz"
  volume_id = "ceph-example"
  name = "ceph-example"
  plugin_id = "cephfs-csi"
  context = {
    clusterID = "090e7e3f-0b34-289q-9202-gh8z35g3db51"
    fsName = "examplefs"
    staticVolume = true
    rootPath = "/volumes/csi/csi-vol-example-084f01c4-9a21-4b5a-a23d-5045e47cb6fz"
  }
  secrets = {
    userID  = "redacted"
    userKey = "redacted"
  }
  capability {
    access_mode     = "multi-node-multi-writer"
    attachment_mode = "file-system"
  }
  mount_options {
    fs_type = "ext4"
  }
}