search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.27k stars 540 forks source link

CephFS CSI Resize feature not working (again ;( #1732

Closed LuckySB closed 3 years ago

LuckySB commented 3 years ago

Describe the bug

Change storage size in pvc from 5G to 7G, pv and pvc object in kubernetes changed

k get pvc -n default
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
csi-cephfs-pvc   Bound    pvc-397b10cf-c546-45e2-9a3d-55a888e55456   7Gi        RWX            csi-cephfs-sc   83m

k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                    STORAGECLASS    REASON   AGE
pvc-397b10cf-c546-45e2-9a3d-55a888e55456   7Gi        RWX            Delete           Bound       default/csi-cephfs-pvc   csi-cephfs-sc            83m

but quota on cephfs directory not changed

getfattr -n ceph.quota.max_bytes 6733a289-beb6-4446-8c3d-c5422b4876f1/
# file: 6733a289-beb6-4446-8c3d-c5422b4876f1/
ceph.quota.max_bytes="5368709120"

Environment details

Steps to reproduce

Steps to reproduce the behavior:

install csi cephfs driver from helm chart 3.1.2 config storage class create pvc, wait for pv change request storage size in pvc - look on cephfs quotas

Actual results

quota on cephfs directory not changed

Expected behavior

Quota must set to 7G

Logs

csi-resizer logs look fine:

I1129 11:03:57.196526       1 controller.go:225] Started PVC processing "default/csi-cephfs-pvc"
I1129 11:03:57.211663       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"397b10cf-c546-45e2-9a3d-55a888e55456", APIVersion:"v1", ResourceVersion:"11514", FieldPath:""}): type: 'Normal' reason: 'Resizing' External resizer is resizing volume pvc-397b10cf-c546-45e2-9a3d-55a888e55456
I1129 11:03:57.219248       1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1129 11:03:57.219261       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":7516192768},"secrets":"***stripped***","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["debug"]}},"access_mode":{"mode":5}},"volume_id":"0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b"}
I1129 11:03:59.035498       1 leaderelection.go:283] successfully renewed lease ceph-csi-cephfs/external-resizer-cephfs-csi-ceph-com
I1129 11:03:59.488737       1 connection.go:185] GRPC response: {"capacity_bytes":7516192768}
I1129 11:03:59.489756       1 connection.go:186] GRPC error: <nil>
I1129 11:03:59.489774       1 controller.go:364] Resize volume succeeded for volume "pvc-397b10cf-c546-45e2-9a3d-55a888e55456", start to update PV's capacity
I1129 11:03:59.503991       1 controller.go:370] Update capacity of PV "pvc-397b10cf-c546-45e2-9a3d-55a888e55456" to 7Gi succeeded
I1129 11:03:59.509733       1 controller.go:399] Resize PVC "default/csi-cephfs-pvc" finished
I1129 11:03:59.509768       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"csi-cephfs-pvc", UID:"397b10cf-c546-45e2-9a3d-55a888e55456", APIVersion:"v1", ResourceVersion:"11514", FieldPath:""}): type: 'Normal' reason: 'VolumeResizeSuccessful' Resize volume succeeded

csi-cephfsplugin log is very interesting !

I1129 11:03:57.222577       1 utils.go:159] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b GRPC call: /csi.v1.Controller/ControllerExpandVolume
I1129 11:03:57.225676       1 utils.go:160] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b GRPC request: {"capacity_range":{"required_bytes":7516192768},"secrets":"***stripped***","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["debug"]}},"access_mode":{"mode":5}},"volume_id":"0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b"}
I1129 11:03:57.756015       1 cephcmds.go:59] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b command succeeded: ceph [-m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 --id fs --keyfile=***stripped*** -c /etc/ceph/ceph.conf fs dump --format=json]
I1129 11:03:58.200630       1 cephcmds.go:59] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b command succeeded: ceph [-m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 --id fs --keyfile=***stripped*** -c /etc/ceph/ceph.conf fs ls --format=json]
I1129 11:03:58.202369       1 omap.go:84] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b got omap values: (pool="cephfs_metadata", namespace="csi", name="csi.volume.1544255a-3232-11eb-bc25-e28d5116787b"): map[csi.imagename:csi-vol-1544255a-3232-11eb-bc25-e28d5116787b csi.volname:pvc-397b10cf-c546-45e2-9a3d-55a888e55456]
I1129 11:03:58.630651       1 cephcmds.go:53] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b an error (exit status 22) and stdError (no valid command found; 10 closest matches:
fs subvolume ls <vol_name> [<group_name>]
fs subvolume create <vol_name> <sub_name> [<size:int>] [<group_name>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>]
fs subvolume rm <vol_name> <sub_name> [<group_name>] [--force]
fs subvolume getpath <vol_name> <sub_name> [<group_name>]
fs subvolume snapshot ls <vol_name> <sub_name> [<group_name>]
fs subvolume snapshot create <vol_name> <sub_name> <snap_name> [<group_name>]
fs subvolume snapshot rm <vol_name> <sub_name> <snap_name> [<group_name>] [--force]
fs subvolume resize <vol_name> <sub_name> <new_size> [<group_name>] [--no-shrink]
fs subvolume snapshot protect <vol_name> <sub_name> <snap_name> [<group_name>]
fs subvolume snapshot unprotect <vol_name> <sub_name> <snap_name> [<group_name>]
Error EINVAL: invalid command
) occurred while running ceph args: [fs subvolume info cephfs csi-vol-1544255a-3232-11eb-bc25-e28d5116787b --group_name csi -m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 -c /etc/ceph/ceph.conf -n client.fs --keyfile=***stripped***]
E1129 11:03:58.630686       1 volume.go:109] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b failed to get subvolume info csi-vol-1544255a-3232-11eb-bc25-e28d5116787b in fs cephfs with Error: an error (exit status 22) and stdError (no valid command found; 10 closest matches:
fs subvolume ls <vol_name> [<group_name>]
fs subvolume create <vol_name> <sub_name> [<size:int>] [<group_name>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>]
fs subvolume rm <vol_name> <sub_name> [<group_name>] [--force]
fs subvolume getpath <vol_name> <sub_name> [<group_name>]
fs subvolume snapshot ls <vol_name> <sub_name> [<group_name>]
fs subvolume snapshot create <vol_name> <sub_name> <snap_name> [<group_name>]
fs subvolume snapshot rm <vol_name> <sub_name> <snap_name> [<group_name>] [--force]
fs subvolume resize <vol_name> <sub_name> <new_size> [<group_name>] [--no-shrink]
fs subvolume snapshot protect <vol_name> <sub_name> <snap_name> [<group_name>]
fs subvolume snapshot unprotect <vol_name> <sub_name> <snap_name> [<group_name>]
Error EINVAL: invalid command
) occurred while running ceph args: [fs subvolume info cephfs csi-vol-1544255a-3232-11eb-bc25-e28d5116787b --group_name csi -m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 -c /etc/ceph/ceph.conf -n client.fs --keyfile=***stripped***]. stdError: no valid command found; 10 closest matches:
fs subvolume ls <vol_name> [<group_name>]
fs subvolume create <vol_name> <sub_name> [<size:int>] [<group_name>] [<pool_layout>] [<uid:int>] [<gid:int>] [<mode>]
fs subvolume rm <vol_name> <sub_name> [<group_name>] [--force]
fs subvolume getpath <vol_name> <sub_name> [<group_name>]
fs subvolume snapshot ls <vol_name> <sub_name> [<group_name>]
fs subvolume snapshot create <vol_name> <sub_name> <snap_name> [<group_name>]
fs subvolume snapshot rm <vol_name> <sub_name> <snap_name> [<group_name>] [--force]
fs subvolume resize <vol_name> <sub_name> <new_size> [<group_name>] [--no-shrink]
fs subvolume snapshot protect <vol_name> <sub_name> <snap_name> [<group_name>]
fs subvolume snapshot unprotect <vol_name> <sub_name> <snap_name> [<group_name>]
Error EINVAL: invalid command
I1129 11:03:59.086690       1 cephcmds.go:59] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b command succeeded: ceph [fs subvolume getpath cephfs csi-vol-1544255a-3232-11eb-bc25-e28d5116787b --group_name csi -m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 -c /etc/ceph/ceph.conf -n client.fs --keyfile=***stripped***]
I1129 11:03:59.487838       1 cephcmds.go:59] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b command succeeded: ceph [fs subvolume create cephfs csi-vol-1544255a-3232-11eb-bc25-e28d5116787b 7516192768 --group_name csi --mode 777 -m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 -c /etc/ceph/ceph.conf -n client.fs --keyfile=***stripped***]
I1129 11:03:59.488396       1 utils.go:165] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b GRPC response: {"capacity_bytes":7516192768}

Additional context

  1. Command fs subvolume info added in 14.2.10, but my version of Ceph is 14.2.9

  2. To change size of cephfs volume driver send command fs subvolume create cephfs, but for my ceph version command should be fs subvolume resize https://github.com/ceph/ceph-csi/issues/1002#issuecomment-624400342

Madhu-1 commented 3 years ago

we will check for subvolume info command it's not present we will fallback for ceph fs subvolume getpath from the logs cephcsi is calling the create command with 7Gib it's expected that cephfs should resize the volume with a new size and as cephfs doesn't require any filesystem resize in node plugin we don't need to worry about it. Have you waited for enough time or tried recreating the application pod or mounting it to a different pod to see that cephfs subvolume never gets resized to never size?

I1129 11:03:59.487838       1 cephcmds.go:59] ID: 23 Req-ID: 0001-0024-ad2326f4-56f8-4c32-9dd3-fa0b20ff2137-0000000000000001-1544255a-3232-11eb-bc25-e28d5116787b command succeeded: ceph [fs subvolume create cephfs csi-vol-1544255a-3232-11eb-bc25-e28d5116787b 7516192768 --group_name csi --mode 777 -m 172.19.212.5:6789,172.19.212.6:6789,172.19.212.7:6789 -c /etc/ceph/ceph.conf -n client.fs --keyfile=***stripped***]
Madhu-1 commented 3 years ago

sorry I overlooked the issue, can you see restarting the cephcsi provisioner pod and try to resize the pvc

LuckySB commented 3 years ago

Thank you for efficiency