Rados Namespace: "failed to create rbd image: rbd: ret=-1, Operation not permitted"

ceph / ceph-csi

CSI driver for Ceph

Apache License 2.0

1.27k stars 537 forks source link

Rados Namespace: "failed to create rbd image: rbd: ret=-1, Operation not permitted" #1705

Closed discostur closed 3 years ago

discostur commented 3 years ago

Hi,

i'm trying to use the new rados namespace feature for RBD images which was introduced in ceph-csi 3.1 release. I already configured a new ceph-csi configmap with the radosNamespace like in the documentation. I also created a new ceph pool with a new user.

As far as i understood from the ceph documentation i cannot create a namespace within a pool - it's just kind of a prefix and gets created automatically when i try to add data to the namespace.

If i try to create rbd-volumes from k8s / ceph-csi i get the following error:

I1118 13:47:40.259809       1 controller.go:562] CreateVolumeRequest {Name:pvc-08368567-417c-4aa6-9216-348f2f8cd918 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode: ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret-ssd csi.storage.k8s.io/controller-expand-secret-namespace:ceph-csi csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret-ssd csi.storage.k8s.io/node-stage-secret-namespace:ceph-csi csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret-ssd csi.storage.k8s.io/provisioner-secret-namespace:ceph-csi imageFeatures:layering imageFormat:2 mounter:rbd-nbd pool:k8s.ssd] Secrets:map[] VolumeContentSource: AccessibilityRequirements: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1118 13:47:40.260158       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"08368567-417c-4aa6-9216-348f2f8cd918", APIVersion:"v1", ResourceVersion:"422256", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pvc"
I1118 13:47:40.262301       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1118 13:47:40.262321       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-08368567-417c-4aa6-9216-348f2f8cd918","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","imageFeatures":"layering","imageFormat":"2","mounter":"rbd-nbd","pool":"k8s.ssd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I1118 13:47:40.287853       1 connection.go:185] GRPC response: {}
I1118 13:47:40.288338       1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1118 13:47:40.288390       1 controller.go:634] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1118 13:47:40.288423       1 controller.go:1051] Final error received, removing PVC 08368567-417c-4aa6-9216-348f2f8cd918 from claims in progress
W1118 13:47:40.288432       1 controller.go:916] Retrying syncing claim "08368567-417c-4aa6-9216-348f2f8cd918", failure 0
E1118 13:47:40.288450       1 controller.go:939] error syncing claim "08368567-417c-4aa6-9216-348f2f8cd918": failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1118 13:47:40.288480       1 controller.go:1284] provision "default/rbd-pvc" class "csi-rbd-ssd": started
I1118 13:47:40.288671       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"08368567-417c-4aa6-9216-348f2f8cd918", APIVersion:"v1", ResourceVersion:"422256", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted

I thought this tells me that my ceph user capabilities are not correct - but i didn't find a proper documentation. I tried a few different settings but none did work:

caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=k8s.ssd namespace=services"

caps mon = "profile rbd"
caps osd = "profile rbd pool=k8s.ssd namespace=services"

Am i right that the error relies on wrong ceph capabilities? If yes, can anyone tell me what capabilities are needed for rbd images with rados namespaces?

Thanks

discostur commented 3 years ago

If i manually look in the pool / namespace i can see that csi has already created one object:

$ rados -p k8s.ssd -N services ls

csi.volumes.default

discostur commented 3 years ago

Just tried with admin credentials - still no luck.

Ceph-CSI is installed via helm chart:

ceph-csi-rbd-3.1.1

Image Versions:

$kubectl -n ceph-csi get deploy ceph-csi-rbd-provisioner -o yaml |grep image:
        image: quay.io/k8scsi/csi-provisioner:v1.6.0
        image: quay.io/k8scsi/csi-resizer:v0.5.0
        image: quay.io/k8scsi/csi-snapshotter:v2.1.0
        image: quay.io/k8scsi/csi-attacher:v2.1.1
        image: quay.io/cephcsi/cephcsi:v3.1.1
        image: quay.io/cephcsi/cephcsi:v3.1.1

discostur commented 3 years ago

Ceph Version:

ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable)

discostur commented 3 years ago

@n-marton do you have any idea? looks like you got it running in https://github.com/ceph/ceph-csi/issues/1397

discostur commented 3 years ago

I investigated a few more hours today ... seems like there is a client side cacheing in ceph-csi? because i just figured out that every time i change ceph authx capabilities via

ceph auth caps ...

i have to restart the ceph-csi pods. Otherwise it looks like it doesn't work. Knowing that, i just did a new test with cluster admin rights:

I1120 15:48:15.615826       1 controller.go:1284] provision "default/rbd-pvc" class "csi-rbd-hdd": started
I1120 15:48:15.621253       1 controller.go:562] CreateVolumeRequest {Name:pvc-08b555ea-f2c7-4142-b751-f44102034ba9 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode: ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace:ceph-csi-rbd csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace:ceph-csi-rbd csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace:ceph-csi-rbd imageFeatures:layering imageFormat:2 pool:k8s.ssd] Secrets:map[] VolumeContentSource: AccessibilityRequirements: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1120 15:48:15.621502       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"08b555ea-f2c7-4142-b751-f44102034ba9", APIVersion:"v1", ResourceVersion:"1214443", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pvc"
I1120 15:48:15.624335       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1120 15:48:15.624354       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-08b555ea-f2c7-4142-b751-f44102034ba9","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","imageFeatures":"layering","imageFormat":"2","pool":"k8s.ssd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I1120 15:48:15.648347       1 connection.go:185] GRPC response: {}
I1120 15:48:15.648779       1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to create rbd image: RBD image not found
I1120 15:48:15.648802       1 controller.go:634] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to create rbd image: RBD image not found
I1120 15:48:15.648847       1 controller.go:1051] Final error received, removing PVC 08b555ea-f2c7-4142-b751-f44102034ba9 from claims in progress
W1120 15:48:15.648858       1 controller.go:916] Retrying syncing claim "08b555ea-f2c7-4142-b751-f44102034ba9", failure 7
E1120 15:48:15.648891       1 controller.go:939] error syncing claim "08b555ea-f2c7-4142-b751-f44102034ba9": failed to provision volume with StorageClass "csi-rbd-hdd": rpc error: code = Internal desc = failed to create rbd image: RBD image not found
I1120 15:48:15.648998       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"08b555ea-f2c7-4142-b751-f44102034ba9", APIVersion:"v1", ResourceVersion:"1214443", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-hdd": rpc error: code = Internal desc = failed to create rbd image: RBD image not found

I also tested the user credentials manually via

$ rados --id k8s.test --keyring ceph.client.k8s.test.keyring -p k8s.ssd -N services put test FILE

and that does work. Also, ceph-csi is able to put some files, because i can see

$ rados --id k8s.test --keyring ceph.client.k8s.test.keyring -p k8s.ssd -N services ls

csi.volumes.default

I also re-deployed without the namespace option and that does work (adjusted user rights before)!

discostur commented 3 years ago

may this be relate to https://github.com/ceph/ceph-csi/pull/1501 ? Thought is is just for the deletion - not for creation ...

Madhu-1 commented 3 years ago

@discostur I will take a look at it soon

discostur commented 3 years ago

@Madhu-1 just saw one more thin which might help you - when i try to create a rbd with cluster admin rights, i can see that the rbd_info file is been created in the wrong default namespace:

$rados -p k8s.ssd ls --all
    rbd_info
services    test
services    csi.volumes.default

csi.volumes.default is in the correct services namespace.

This makes also sense because with the normal credentials the user is NOT allowed to write into the default namespace and then i get the operation not allowed error. With admin credentials, the rbd_info file is created in the wrong namespace and i get rbd not found...

MrOffline77 commented 3 years ago

Hi, I'm also interested in a solution for this.

Madhu-1 commented 3 years ago

I1123 08:37:33.388010       1 utils.go:160] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad","parameters":{"clusterID":"rook-ceph","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I1123 08:37:33.389056       1 rbd_util.go:771] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad setting disableInUseChecks on rbd volume to: false
I1123 08:37:33.389938       1 omap.go:72] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad got omap values: (pool="replicapool", namespace="testing", name="csi.volumes.default"): map[]
I1123 08:37:33.415505       1 omap.go:136] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad set omap keys (pool="replicapool", namespace="testing", name="csi.volumes.default"): map[csi.volume.pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad:219f47b2-2d67-11eb-a0b9-0242ac110006])
I1123 08:37:33.426156       1 omap.go:136] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad set omap keys (pool="replicapool", namespace="testing", name="csi.volume.219f47b2-2d67-11eb-a0b9-0242ac110006"): map[csi.imagename:csi-vol-219f47b2-2d67-11eb-a0b9-0242ac110006 csi.volname:pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad])
I1123 08:37:33.426178       1 rbd_journal.go:443] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad generated Volume ID (0001-0009-rook-ceph-0000000000000002-219f47b2-2d67-11eb-a0b9-0242ac110006) and image name (csi-vol-219f47b2-2d67-11eb-a0b9-0242ac110006) for request name (pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad)
I1123 08:37:33.426241       1 rbd_util.go:198] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad rbd: create replicapool/testing/csi-vol-219f47b2-2d67-11eb-a0b9-0242ac110006 size 1024M (features: [layering]) using mon 10.103.251.139:6789
E1123 08:37:33.441708       1 controllerserver.go:454] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad failed to create volume: failed to create rbd image: RBD image not found
I1123 08:37:33.462772       1 omap.go:106] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad removed omap keys (pool="replicapool", namespace="testing", name="csi.volumes.default"): [csi.volume.pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad]
E1123 08:37:33.462892       1 utils.go:163] ID: 32 Req-ID: pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad GRPC error: rpc error: code = Internal desc = failed to create rbd image: RBD image not found

If you are seeing failed to create rbd image: RBD image not found error means the namespace doesn't exist in ceph, to validate it run the below command in your ceph cluster

sh-4.4# rbd namespace ls --pool=replicapool

If the output is empty, the admin Need to create the namespace,cephcsi will not create the rbd namespace

sh-4.4# rbd namespace create --pool=replicapool --namespace=testing
sh-4.4# rbd namespace ls --pool=replicapool
NAME   
testing

Once I create the rbd namespace my PVC went to Bound state.

[🎩︎]mrajanna@localhost csi $]kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
rbd-pvc   Bound    pvc-8fc48743-0950-48a5-97f2-1c990c1e82ad   1Gi        RWO            rook-ceph-block   6m20s

sh-4.4# rbd ls --pool=replicapool --namespace=testing
csi-vol-6dfc44e0-2d67-11eb-a0b9-0242ac110006

discostur commented 3 years ago

@Madhu-1 thanks for the hint with the manual creation of the namespace. I can confirm that if i manually create the namespace i am able to create a PVC. Maybe you should write this to the documentation - it was not clear for me. Sorry for that ;)

But one more question: the ceph user still needs rights to access the default namespace. Is this correct? I thought this feature is for multi tenancy so in my opinion it does not make any sense that every user needs to have access to the default namespace?

I just tested this out:

create ceph user with rwx rights for the namespace:

$ ceph auth caps client.k8stest mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=k8s.ssd namespace=services'

create rbd namespace:

$ rbd namespace create --pool=k8s.ssd --namespace services

$ rados -p k8s.ssd ls --all
    rbd_namespace
services    rbd_directory

$ rbd namespace ls --pool=k8s.ssd
NAME                 
services

When i now try to create a PVC i get the "Permission Denied" error. If i update the user and give him access to the default namspace / pool

$ ceph auth caps client.k8stest mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=k8s.ssd namespace=services, allow rwx pool=k8s.ssd'

PVC creation works. I also tried it with just read rights to the pool / default namespace but that doesn't work.

Seems like the rbd_info object is the only thing which is placed into the default namespace:

$ rados -p k8s.ssd ls --all
    rbd_namespace
    rbd_info
services    rbd_header.8c8596484f26dc
services    rbd_directory
services    csi.volumes.default
services    csi.volume.612bdde0-2d6b-11eb-866a-c20151b73ef7
services    rbd_id.csi-vol-612bdde0-2d6b-11eb-866a-c20151b73ef7

Why is rbd_info still in the default namespace and not in the correct services namespace? If i have multiple namespaces for multiple users - does it make sense to put one object still in a shared default namespace which every user needs access to? Or do i understand anything wrong?

Thanks

Madhu-1 commented 3 years ago

cephcsi won't create rbd_info, let me check that one, we still need to write a document and add E2E testing for the user caps

discostur commented 3 years ago

@Madhu-1 while you are trying to figure out where the rbd_info file is comeing from ... can you confirm that RHEL7 has no support for mounting rbd images in specific namespaces?

I just tried to mount it via krbd but

[ 1542.266688] libceph: bad option at '_pool_ns=services'

looks like the RHEL7.9 kernel (krbd) does still not support rbd namespaces. rbd-nbd seems neither to be an option, because the needed nbd kernel module is missing in RHEL 7.9 :/

$ modprobe nbd
modprobe: FATAL: Module nbd not found.

Kernel Version: 3.10.0-1160.6.1.el7.x86_64

Needed krbd kernel support: https://github.com/torvalds/linux/commit/b26c047b940003295d3896b7f633a66aab95bebd

Thanks

discostur commented 3 years ago

Meanwhile i tested both, krbd and rbd-nbd with RHEL8 / CentOS8 and it does both work (out of the box).

Madhu-1 commented 3 years ago

@pkalever can I assign this to you, we need to update our E2E to test with the namespaced user. https://github.com/ceph/ceph-csi/blob/master/e2e/rbd.go#L1077-L1169 , currently we are using user who is having access to all pools.

clwluvw commented 3 years ago

It seems there is a need to use kernel 4.19+ for this feature! http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034878.html

Madhu-1 commented 3 years ago

@discostur I tried it today on my setup being able to create the rbd image with restricted user caps

cephcsi v3.2.0 version used for testing

Provisioner logs

I1218 13:02:10.005983       1 utils.go:132] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 GRPC call: /csi.v1.Controller/CreateVolume
I1218 13:02:10.006772       1 utils.go:133] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-97e37f33-5b5b-4fa8-a8b6-069593044094","parameters":{"clusterID":"rook-ceph","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I1218 13:02:10.007344       1 rbd_util.go:808] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 setting disableInUseChecks on rbd volume to: false
E1218 13:02:10.027243       1 omap.go:77] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 omap not found (pool="replicapool", namespace="test", name="csi.volumes.default"): rados: ret=-2, No such file or directory
I1218 13:02:10.037518       1 omap.go:148] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 set omap keys (pool="replicapool", namespace="test", name="csi.volumes.default"): map[csi.volume.pvc-97e37f33-5b5b-4fa8-a8b6-069593044094:3d29ad27-4131-11eb-b260-0242ac110003])
I1218 13:02:10.042708       1 omap.go:148] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 set omap keys (pool="replicapool", namespace="test", name="csi.volume.3d29ad27-4131-11eb-b260-0242ac110003"): map[csi.imagename:csi-vol-3d29ad27-4131-11eb-b260-0242ac110003 csi.volname:pvc-97e37f33-5b5b-4fa8-a8b6-069593044094])
I1218 13:02:10.042888       1 rbd_journal.go:436] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 generated Volume ID (0001-0009-rook-ceph-0000000000000002-3d29ad27-4131-11eb-b260-0242ac110003) and image name (csi-vol-3d29ad27-4131-11eb-b260-0242ac110003) for request name (pvc-97e37f33-5b5b-4fa8-a8b6-069593044094)
I1218 13:02:10.043122       1 rbd_util.go:201] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 rbd: create replicapool/test/csi-vol-3d29ad27-4131-11eb-b260-0242ac110003 size 1024M (features: [layering]) using mon 10.96.137.117:6789
I1218 13:02:10.064102       1 controllerserver.go:486] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 created volume pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 backed by image csi-vol-3d29ad27-4131-11eb-b260-0242ac110003
I1218 13:02:10.089421       1 omap.go:148] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 set omap keys (pool="replicapool", namespace="test", name="csi.volume.3d29ad27-4131-11eb-b260-0242ac110003"): map[csi.imageid:5f26448e9737])
I1218 13:02:10.089848       1 utils.go:138] ID: 20 Req-ID: pvc-97e37f33-5b5b-4fa8-a8b6-069593044094 GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"rook-ceph","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-3d29ad27-4131-11eb-b260-0242ac110003","journalPool":"replicapool","pool":"replicapool","radosNamespace":"test"},"volume_id":"0001-0009-rook-ceph-0000000000000002-3d29ad27-4131-11eb-b260-0242ac110003"}}

ceph version

sh-4.4# ceph version
ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)

RBD image list from ceph cluster

sh-4.4# rbd ls --pool=replicapool
sh-4.4# rbd ls --pool=replicapool --namespace=test
csi-vol-3d29ad27-4131-11eb-b260-0242ac110003

User caps

# ceph user caps
client.csi-rbd-provisioner
key: AQD+JNtf0qNwBxAAnQcCoq3bpZM+iRRHxwt5ew==
caps: [mgr] allow rw
caps: [mon] profile rbd
caps: [osd] profile rbd pool=replicapool namespace=test

discostur commented 3 years ago

@Madhu-1 i just did another test with the latest ceph-csi v3.2.0 but for me it is still only working with the following user caps:

    caps mgr = "allow rw"
    caps mon = "profile rbd"
    caps osd = "profile rbd pool=k8s.ssd namespace=services, profile rbd pool=k8s.ssd"

If i try it with the one you suggested

    caps mgr = "allow rw"
    caps mon = "profile rbd"
    caps osd = "profile rbd pool=k8s.ssd namespace=services"

i still get the operation not permitted error:

I1222 15:33:04.354577       1 controller.go:1317] provision "default/rbd-pvc" class "csi-rbd-ssd": started
I1222 15:33:04.354680       1 controller.go:573] CreateVolumeRequest {Name:pvc-ff14d0ac-2094-44b4-af94-b83793318f29 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode: ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace:default csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace:default csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace:default imageFeatures:layering pool:k8s.ssd] Secrets:map[] VolumeContentSource: AccessibilityRequirements: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1222 15:33:04.355002       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"ff14d0ac-2094-44b4-af94-b83793318f29", APIVersion:"v1", ResourceVersion:"9959695", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pvc"
I1222 15:33:04.358841       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1222 15:33:04.358859       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-ff14d0ac-2094-44b4-af94-b83793318f29","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","imageFeatures":"layering","pool":"k8s.ssd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I1222 15:33:04.378236       1 connection.go:185] GRPC response: {}
I1222 15:33:04.378268       1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1222 15:33:04.378294       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1222 15:33:04.378327       1 controller.go:1084] Final error received, removing PVC ff14d0ac-2094-44b4-af94-b83793318f29 from claims in progress
W1222 15:33:04.378339       1 controller.go:943] Retrying syncing claim "ff14d0ac-2094-44b4-af94-b83793318f29", failure 5
E1222 15:33:04.378360       1 controller.go:966] error syncing claim "ff14d0ac-2094-44b4-af94-b83793318f29": failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1222 15:33:04.378393       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"ff14d0ac-2094-44b4-af94-b83793318f29", APIVersion:"v1", ResourceVersion:"9959695", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted

The only thing i know which is different form your setup is our ceph version:

ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable)

Could it be that there was a change in the ceph version and that's why i need extended user caps than you? Maybe in the profile rbd caps?

deadjoker commented 3 years ago

same error when I use ceph-csi provide cephfs pvc. I think it's the caps issue because the pvc was created succcessfully when I use admin key. Is there a minimum caps for ceph-csi cephfs user? this is my user caps:

client.k8sfs
    key: AQDuM+xfXz0zNRAAnxeJaWdmR2J5I/QxMR9gLQ==
    caps: [mds] allow rwps
    caps: [mgr] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw tag cephfs data=cephfs

Madhu-1 commented 3 years ago

@deadjoker please refer to https://github.com/ceph/ceph-csi/blob/master/docs/capabilities.md#cephfs

deadjoker commented 3 years ago

@Madhu-1 thanks for the replay. I modify the caps according to the docs and I get the same error. the new caps:

client.k8sfs
    key: AQDuM+xfXz0zNRAAnxeJaWdmR2J5I/QxMR9gLQ==
    caps: [mds] allow rwps
    caps: [mgr] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw tag cephfs *=*

Madhu-1 commented 3 years ago

caps: [mds] allow rwps

this should be rw, except that everything looks good. we use the same caps, everything works for us.

deadjoker commented 3 years ago

I use '[mds] allow rw' and has no luck as well. the p and s flag only add caps to set quota and make snapshot. provisioner log:

 I1230 09:09:38.293338       1 controller.go:1317] provision "default/csi-cephfs-pvc" class "csi-cephfs-sc": started                                                                                                                                             │
│ I1230 09:09:38.293602       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"4bd0ecf9-d613-4f8f-998d-d0b204e8352d", APIVersion:"v1", ResourceVersion:"1814335", FieldPath:""}): type: 'N │
│ ormal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/csi-cephfs-pvc"                                                                                                                                                    │
│ I1230 09:09:38.293516       1 controller.go:573] CreateVolumeRequest {Name:pvc-4bd0ecf9-d613-4f8f-998d-d0b204e8352d CapacityRange:required_bytes:5368709120  VolumeCapabilities:[mount:<mount_flags:"debug" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Par │
│ ameters:map[clusterID:d9693b9b-8988-44bb-8bf9-ccb2c2733eec csi.storage.k8s.io/controller-expand-secret-name:csi-cephfs-secret csi.storage.k8s.io/controller-expand-secret-namespace:ceph-csi csi.storage.k8s.io/node-stage-secret-name:csi-cephfs-secret csi.st │
│ orage.k8s.io/node-stage-secret-namespace:ceph-csi csi.storage.k8s.io/provisioner-secret-name:csi-cephfs-secret csi.storage.k8s.io/provisioner-secret-namespace:ceph-csi fsName:cephfs mounter:kernel] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequ │
│ irements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}                                                                                                                                                                                     │
│ I1230 09:09:38.304983       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume                                                                                                                                                                     │
│ I1230 09:09:38.305017       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-4bd0ecf9-d613-4f8f-998d-d0b204e8352d","parameters":{"clusterID":"d9693b9b-8988-44bb-8bf9-ccb2c2733eec","fsName":"cephfs","mounter":" │
│ kernel"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["debug"]}},"access_mode":{"mode":5}}]}                                                                                                                        │
│ I1230 09:09:38.309832       1 connection.go:185] GRPC response: {}                                                                                                                                                                                              │
│ I1230 09:09:38.309894       1 connection.go:186] GRPC error: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted                                                                                                                           │
│ I1230 09:09:38.309949       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted                    │
│ I1230 09:09:38.310012       1 controller.go:1084] Final error received, removing PVC 4bd0ecf9-d613-4f8f-998d-d0b204e8352d from claims in progress                                                                                                               │
│ W1230 09:09:38.310036       1 controller.go:943] Retrying syncing claim "4bd0ecf9-d613-4f8f-998d-d0b204e8352d", failure 8                                                                                                                                       │
│ E1230 09:09:38.310070       1 controller.go:966] error syncing claim "4bd0ecf9-d613-4f8f-998d-d0b204e8352d": failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted             │
│ I1230 09:09:38.310134       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"4bd0ecf9-d613-4f8f-998d-d0b204e8352d", APIVersion:"v1", ResourceVersion:"1814335", FieldPath:""}): type: 'W │
│ arning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted

Madhu-1 commented 3 years ago

I use '[mds] allow rw' and has no luck as well. the p and s flag only add caps to set quota and make snapshot. provisioner log:

 I1230 09:09:38.293338       1 controller.go:1317] provision "default/csi-cephfs-pvc" class "csi-cephfs-sc": started                                                                                                                                             │
│ I1230 09:09:38.293602       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"4bd0ecf9-d613-4f8f-998d-d0b204e8352d", APIVersion:"v1", ResourceVersion:"1814335", FieldPath:""}): type: 'N │
│ ormal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/csi-cephfs-pvc"                                                                                                                                                    │
│ I1230 09:09:38.293516       1 controller.go:573] CreateVolumeRequest {Name:pvc-4bd0ecf9-d613-4f8f-998d-d0b204e8352d CapacityRange:required_bytes:5368709120  VolumeCapabilities:[mount:<mount_flags:"debug" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Par │
│ ameters:map[clusterID:d9693b9b-8988-44bb-8bf9-ccb2c2733eec csi.storage.k8s.io/controller-expand-secret-name:csi-cephfs-secret csi.storage.k8s.io/controller-expand-secret-namespace:ceph-csi csi.storage.k8s.io/node-stage-secret-name:csi-cephfs-secret csi.st │
│ orage.k8s.io/node-stage-secret-namespace:ceph-csi csi.storage.k8s.io/provisioner-secret-name:csi-cephfs-secret csi.storage.k8s.io/provisioner-secret-namespace:ceph-csi fsName:cephfs mounter:kernel] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequ │
│ irements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}                                                                                                                                                                                     │
│ I1230 09:09:38.304983       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume                                                                                                                                                                     │
│ I1230 09:09:38.305017       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-4bd0ecf9-d613-4f8f-998d-d0b204e8352d","parameters":{"clusterID":"d9693b9b-8988-44bb-8bf9-ccb2c2733eec","fsName":"cephfs","mounter":" │
│ kernel"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["debug"]}},"access_mode":{"mode":5}}]}                                                                                                                        │
│ I1230 09:09:38.309832       1 connection.go:185] GRPC response: {}                                                                                                                                                                                              │
│ I1230 09:09:38.309894       1 connection.go:186] GRPC error: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted                                                                                                                           │
│ I1230 09:09:38.309949       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted                    │
│ I1230 09:09:38.310012       1 controller.go:1084] Final error received, removing PVC 4bd0ecf9-d613-4f8f-998d-d0b204e8352d from claims in progress                                                                                                               │
│ W1230 09:09:38.310036       1 controller.go:943] Retrying syncing claim "4bd0ecf9-d613-4f8f-998d-d0b204e8352d", failure 8                                                                                                                                       │
│ E1230 09:09:38.310070       1 controller.go:966] error syncing claim "4bd0ecf9-d613-4f8f-998d-d0b204e8352d": failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted             │
│ I1230 09:09:38.310134       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"4bd0ecf9-d613-4f8f-998d-d0b204e8352d", APIVersion:"v1", ResourceVersion:"1814335", FieldPath:""}): type: 'W │
│ arning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted

can you check csi-cephfsplugin container for now detailed error. and also can you try to use the same user to create cephfs subvolume and subvolumegroup manually in ceph cluster?

Madhu-1 commented 3 years ago

@Madhu-1 i just did another test with the latest ceph-csi v3.2.0 but for me it is still only working with the following user caps:

  caps mgr = "allow rw"
  caps mon = "profile rbd"
  caps osd = "profile rbd pool=k8s.ssd namespace=services, profile rbd pool=k8s.ssd"

If i try it with the one you suggested

  caps mgr = "allow rw"
  caps mon = "profile rbd"
  caps osd = "profile rbd pool=k8s.ssd namespace=services"

i still get the operation not permitted error:

I1222 15:33:04.354577       1 controller.go:1317] provision "default/rbd-pvc" class "csi-rbd-ssd": started
I1222 15:33:04.354680       1 controller.go:573] CreateVolumeRequest {Name:pvc-ff14d0ac-2094-44b4-af94-b83793318f29 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode: ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace:default csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace:default csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace:default imageFeatures:layering pool:k8s.ssd] Secrets:map[] VolumeContentSource: AccessibilityRequirements: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1222 15:33:04.355002       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"ff14d0ac-2094-44b4-af94-b83793318f29", APIVersion:"v1", ResourceVersion:"9959695", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/rbd-pvc"
I1222 15:33:04.358841       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1222 15:33:04.358859       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-ff14d0ac-2094-44b4-af94-b83793318f29","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","imageFeatures":"layering","pool":"k8s.ssd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I1222 15:33:04.378236       1 connection.go:185] GRPC response: {}
I1222 15:33:04.378268       1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1222 15:33:04.378294       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1222 15:33:04.378327       1 controller.go:1084] Final error received, removing PVC ff14d0ac-2094-44b4-af94-b83793318f29 from claims in progress
W1222 15:33:04.378339       1 controller.go:943] Retrying syncing claim "ff14d0ac-2094-44b4-af94-b83793318f29", failure 5
E1222 15:33:04.378360       1 controller.go:966] error syncing claim "ff14d0ac-2094-44b4-af94-b83793318f29": failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted
I1222 15:33:04.378393       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"rbd-pvc", UID:"ff14d0ac-2094-44b4-af94-b83793318f29", APIVersion:"v1", ResourceVersion:"9959695", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd-ssd": rpc error: code = Internal desc = failed to create rbd image: rbd: ret=-1, Operation not permitted

The only thing i know which is different form your setup is our ceph version:

ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable)

Could it be that there was a change in the ceph version and that's why i need extended user caps than you? Maybe in the profile rbd caps?

The above one gives access to the pool, we want to restrict the user for the namespace, Yes the ceph cluster version is the only difference, let me try out and see is there any issue with ceph 1.14.x

deadjoker commented 3 years ago

The 'k8sfs' user can create subvolume and subvolumegroup.

[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolume create cephfs test 
[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolume ls cephfs
[
    {
        "name": "test"
    }
]
[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolumegroup create cephfs testgroup
[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolumegroup ls cephfs
[
    {
        "name": "_nogroup"
    }, 
    {
        "name": "csi"
    }, 
    {
        "name": "testgroup"
    }
]

The 'csi' subvolumegroup is created when I use admin keyring in ceph-csi.

Madhu-1 commented 3 years ago

The 'k8sfs' user can create subvolume and subvolumegroup.

[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolume create cephfs test 
[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolume ls cephfs
[
    {
        "name": "test"
    }
]
[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolumegroup create cephfs testgroup
[root@ceph-node1 ceph]# ceph --id k8sfs fs subvolumegroup ls cephfs
[
    {
        "name": "_nogroup"
    }, 
    {
        "name": "csi"
    }, 
    {
        "name": "testgroup"
    }
]

The 'csi' subvolumegroup is created when I use admin keyring in ceph-csi.

okay, am not sure what is the issue here, can you create a subvolume using k8sfs user in csi subvolumegroup?. can you please open a new issue, can we not discuss the cephfs issue here.

deadjoker commented 3 years ago

@Madhu-1 refer to https://github.com/ceph/ceph-csi/issues/1818

discostur commented 3 years ago

@Madhu-1 i did a huge debug session today and i re-created a new and untouched ceph octopus cluster v15.2.8. Sadly, the results are still the same. However, i found out that if i first create one rbd image in a namespace (with admin user) and then limit the user rights, all is working fine - even when i delete the namespace and then re-create it. So the problem only occurs with the first rbd image creation (per pool)!

If i then delete the pool and re-create everything i get the error again:

- create pool
- create namespace
- create ohne RBD image inside namespace with admin user
-> create RBD image with limited user rights is working

- create pool (anotherone, or delete the first pool and re-create it)
- create namespace
-> create RBD image with limited user rights is NOT working

So the problem is NOT related to the ceph version.

That brings me back to the beginning of this ticket where i have mentioned that one file will be created outside of the namespace - directly to the pool:

https://github.com/ceph/ceph-csi/issues/1705#issuecomment-731266753

Can you please re-check on your test environment that this file is also being created? Does it work for you on a completly new pool?

Can you please investigate where the files "rbd_info" and "csi.volumes.default" are coming from? It seems to me that this is still the root of the problem ...

just after pool / namespace creation (still the same with limited user rights)

$rados -p ix.k8s-test ls --all
    rbd_namespace
shared-services rbd_directory
shared-services csi.volumes.default

after rbd image creation with admin user

$rados -p ix.k8s-test ls --all
    rbd_namespace
    rbd_info
shared-services rbd_directory
    csi.volumes.default
shared-services csi.volumes.default
shared-services csi.volume.7a111dcc-59ba-11eb-a288-96e8fca18cc2
shared-services rbd_trash

Thanks

discostur commented 3 years ago

ping @Madhu-1

Madhu-1 commented 3 years ago

@discostur am adding e2e to test rados namespace with limited caps user in https://github.com/ceph/ceph-csi/pull/1888

Madhu-1 commented 3 years ago

csi.volumes.default"

@discostur is created by the cephcsi driver.

Madhu-1 commented 3 years ago

@discostur I can create pvc inside the rbd namespace with a restricted user with caps mentioned at https://github.com/ceph/ceph-csi/issues/1705#issuecomment-748075784. E2E is passing in https://github.com/ceph/ceph-csi/pull/1888 please check if am missing something or am testing it differently.

Madhu-1 commented 3 years ago

one last thing I need to test is to create a pvc in the rbd namespace first without creating anything to make sure we don't have any issue in rados omap.

Madhu-1 commented 3 years ago

Closing this one, Due to inactivity