Madhu-1
commented
1 year ago @rimkov You can check with the cephfs team in Slack at https://ceph-storage.slack.com/
Closed rimkov closed 1 year ago
Madhu-1
commented
1 year ago @rimkov You can check with the cephfs team in Slack at https://ceph-storage.slack.com/
rimkov
commented
1 year ago oh thank you for this and sorry for the noise.
Describe the bug
I'm sorry, it's not a bug related to the csi driver but nobody gave me an answer on the ceph-user list, I hope to have a start here. I have a pool named "data" and a fs "kube"
the creation of subvolume worked perfectly until a few days ago, now it gives me the following error :
Error EINVAL: invalid value specified for ceph.dir.subvolumethe creation of the pvc is then pending.
I tried to create a subvolume directly from the command line and I got the same error message.
ceph fs subvolume create kube newcsivol --pool_layout dataI then activated the debug mode for mgr and mds and I could see this in the logs:
2023-06-23T08:30:53.307+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] Starting _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], vol_name:kube) < "" 2023-06-23T08:30:53.327+0000 7f2b8a1d1700 0 [volumes INFO volumes.fs.operations.versions.subvolume_v2] cleaning up subvolume with path: newcsivol 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO volumes.fs.operations.versions.subvolume_base] subvolume path 'b'/volumes/csi/newcsivol'' moved to trashcan 2023-06-23T08:30:53.331+0000 7f2b8a1d1700 0 [volumes INFO volumes.fs.async_job] queuing job for volume 'kube' 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 0 [volumes INFO volumes.module] Finishing _cmd_fs_subvolume_create(group_name:csi, pool_layout:data, prefix:fs subvolume create, sub_name:newcsivol, target:['mon-mgr', ''], vol_name:kube) < "" 2023-06-23T08:30:53.335+0000 7f2b8a1d1700 -1 mgr.server reply reply (22) Invalid argument invalid value specified for ceph.dir.subvolumeit seems that the new subvolume created is sent directly to the trash.
if I create a new fs I have no problem creating a sub volume
thanks a lot for your help
Environment details
fuseorkernel. for rbd itskrbdorrbd-nbd) : not specifiedExpected behavior
subvolume created successfully
Logs
here are the logs of the ceph-csi-cephfs-provisioner container when creating a pvc named "dd":
I0628 08:59:22.892080 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:22.892413 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:22.955397 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 0 I0628 08:59:22.955579 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" E0628 08:59:22.956125 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:23.457033 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:23.457277 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:23.513582 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 1 E0628 08:59:23.513614 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:23.513740 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:24.513970 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:24.514668 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:24.573781 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 2 E0628 08:59:24.573870 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:24.573900 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:26.586788 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:26.587166 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:26.646305 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 3 I0628 08:59:26.646369 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" E0628 08:59:26.646831 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:30.647251 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:30.647849 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:30.698464 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 4 E0628 08:59:30.698535 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:30.698580 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:38.699330 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:38.700044 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:38.755664 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 5 E0628 08:59:38.755706 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:38.756076 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:54.756216 1 controller.go:1337] provision "castopod/dd" class "ceph-sc-delete": started I0628 08:59:54.757083 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "castopod/dd" W0628 08:59:54.910879 1 controller.go:934] Retrying syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4", failure 6 E0628 08:59:54.910928 1 controller.go:957] error syncing claim "52a620a3-fec2-4118-9aca-3cb9b5314cc4": failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume" I0628 08:59:54.910968 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"castopod", Name:"dd", UID:"52a620a3-fec2-4118-9aca-3cb9b5314cc4", APIVersion:"v1", ResourceVersion:"148061417", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ceph-sc-delete": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "invalid value specified for ceph.dir.subvolume"