crltc commented 7 months ago

Describe the bug

external_id generated by plugin uses incorrect pool-id for cephfs volumes. The cephfs volumes I am deploying are designated to pool 21 and 22 but ceph-csi provisions it as pool 4 which does not exist. Mounting cephfs requires the use of pool-id 4 instead of the actual pool id.

external_id consists of: <csi-id-version>-<cluster-id-length>-<cluster-id>-<pool-id>-<uuid>

Environment details

Image/version of Ceph CSI driver: quay.io/cephcsi/cephcsi:v3.10.2
Nomad version: 1.7.5
Kernel version : 6.1.0-18-amd64
Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its krbd or rbd-nbd) : kernel
Ceph cluster version : 18.2.1 reef (stable)

Steps to reproduce

Setup Details: I have a near identical setup to the example provided here: https://docs.ceph.com/en/latest/rbd/rbd-nomad/

Deployment: Run nomad volume job with Terraform to create volume:

resource "nomad_csi_volume" "example" {
  volume_id = "ceph-example"
  name = "ceph-example"
  plugin_id = "cephfs-csi"
  context = {
    clusterID = "090e7e3f-0b34-289q-9202-gh8z35g3db51"
    fsName = "examplefs"
  }
  secrets = {
    adminID  = "redacted"
    adminKey = "redacted"
  }
  capability {
    access_mode     = "multi-node-multi-writer"
    attachment_mode = "file-system"
  }
  mount_options {
    fs_type = "ext4"
  }
}

Actual results

Deployed pool within external_id has pool-id 4 even though my cluster does not have a pool with id 4

0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000004-084f01c4-9a21-4b5a-a23d-5045e47cb6fz

Expected behavior

Deployed pool should have pool-id 21 within it's external_id.

0001-0024-090e7e3f-0b34-289q-9202-gh8z35g3db51-0000000000000015-084f01c4-9a21-4b5a-a23d-5045e47cb6fz

Logs

Output from ceph osd lspools:

1 .mgr
2 ceph
5 ceph-ssd
7 .rgw.root
8 default.rgw.log
9 default.rgw.control
10 default.rgw.meta
12 examplerbd
21 examplefs_data
22 examplefs_metadata

Madhu-1 commented 7 months ago

@crltc we dont use pool-id for cephfs volumes we use filesystem ID.