Can Ceph CSI driver use to multi ceph cluster?

[andyzheung]

As the title said, I create a ceph cluster outside the k8s cluster using rook. And I want to use cephfs csi driver to driver the two ceph like the following:

my csi-config-map.yaml

apiVersion: v1 kind: ConfigMap data: config.json: |- [ { "clusterID": "61ef19c8-4076-46c1-b669-09ae77fcf9a0", "monitors": [ "10.240.62.11:6789", "10.240.62.12:6789", "10.240.62.13:6789" ], "cephFS": { "subvolumeGroup": "multi-dev" } }, { "clusterID": "fe00179f-2e0c-49d3-9a3e-97001bf5270e", "monitors": [ "10.250.68.246:6789", "10.250.68.246:6790", "10.250.68.246:6791" ], "cephFS": { "subvolumeGroup": "privatecloud-rook-dev" } } ] metadata: name: ceph-csi-config namespace: csi-cephfs

ceph seceret 1:

apiVersion: v1 kind: Secret metadata: name: csi-cephfs-secret namespace: csi-cephfs stringData:

userID: admin userKey: AQCRQ1BhPUEpMhAAAn8+TlhjF9XIOYfqULOt8Q==

adminID: admin adminKey: AQCRQ1BhPUEpMhAAAn8+TlhjF9XIOYfqULOt8Q==

ceph seceret 2:

apiVersion: v1 kind: Secret metadata: name: rook-csi-cephfs-secret namespace: csi-cephfs stringData:

userID: cephfs userKey: AQA/Ajpko7qCHhAAxsgEX7spT1xKFz2HBvFpLw==

adminID: admin adminKey: AQDddjlkj+REKBAAqkugRQvFLIQDzDpLnrlJpw==

kubectl get pvc -n demo csi-cephfs-pvc Bound pvc-216c4e60-ad1d-4eb5-8cbb-6ff7883919bf 1Gi RWX multi-dev-csi-cephfs-sc 3h22m si-rook-cephfs-pvc Pending dev-csi-rook-cephfs-sc 60s

rook ceph storage: cat rook-cephfs-storageclass.yaml

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: dev-csi-rook-cephfs-sc provisioner: cephfs.csi.ceph.com parameters: clusterID: fe00179f-2e0c-49d3-9a3e-97001bf5270e

fsName: cephfs

pool: cephfs_data

csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-secret csi.storage.k8s.io/provisioner-secret-namespace: csi-cephfs csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-secret csi.storage.k8s.io/controller-expand-secret-namespace: csi-cephfs csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-secret csi.storage.k8s.io/node-stage-secret-namespace: csi-cephfs

reclaimPolicy: Delete allowVolumeExpansion: true

cephfs provisioner pod logs like this: W0415 07:26:40.584279 1 controller.go:934] Retrying syncing claim "606cd33f-aad0-4176-8899-34ec20b53e6a", failure 6 E0415 07:26:40.584302 1 controller.go:957] error syncing claim "606cd33f-aad0-4176-8899-34ec20b53e6a": failed to provision volume with StorageClass "dev-csi-rook-cephfs-sc": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-606cd33f-aad0-4176-8899-34ec20b53e6a already exists I0415 07:26:40.584311 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"demo", Name:"csi-rook-cephfs-pvc", UID:"606cd33f-aad0-4176-8899-34ec20b53e6a", APIVersion:"v1", ResourceVersion:"1324078", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "dev-csi-rook-cephfs-sc": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-606cd33f-aad0-4176-8899-34ec20b53e6a already exists

[andyzheung]

except the provisoner pod log, don't have any other error logs. Is the secret error or network error?

[Madhu-1]

am closing this one, let's continue on the above Rook issue.

[zentavr]

So how to achieve this?