nickatnceas
commented
11 months ago I made the following changes:
Created a k8s-dev-cephfs user in Ceph with appropriate permissions:
ceph auth get-or-create client.k8s-dev-cephfs mon 'allow r' osd 'allow rw tag cephfs *=*' mgr 'allow rw' mds 'allow rw'Changed userID and userKey to adminID and adminKey in secret csi-cephfs-secret, using the new k8s-dev-cephfs Ceph user:
kubectl edit secrets csi-cephfs-secretChanged clusterID to our cluster UUID, and fsName to our CephFS name, in storageclass csi-cephfs-sc
kubectl get storageclass csi-cephfs-sc -o yaml
vim csi-cephfs-sc.yaml
kubectl replace -f csi-cephfs-sc.yaml --forceI expected this to allow me to create a CephFS PVC and PV, but it continually failed:
cephfs-pvs.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cephfs-pvc-test-4
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: csi-cephfs-scoutin@halt:~/k8s$ kubectl create -f cephfs-pvc.yaml -n nick
outin@halt:~/k8s$ kubectl describe pvc -n nick csi-cephfs-pvc-test-4
Name: csi-cephfs-pvc-test-4
Namespace: nick
StorageClass: csi-cephfs-sc
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 109s (x11 over 10m) cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq_ff8d4afe-2849-4ff7-928b-b0f7a8f171dc External provisioner is provisioning volume for claim "nick/csi-cephfs-pvc-test-4"
Warning ProvisioningFailed 109s (x11 over 10m) cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq_ff8d4afe-2849-4ff7-928b-b0f7a8f171dc failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = InvalidArgument desc = failed to get connection: connecting failed: rados: ret=-13, Permission denied
Normal ExternalProvisioning 8s (x103 over 25m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cephfs.csi.ceph.com" or manually created by system administratorI tried changing the adminID and adminKey to the Ceph admin user, and it still failed, which rules out a permissions caps issue in Ceph and received the same error message
I tried restarting the three cephfs-provisioner pods, did not help:
kubectl -n ceph-csi-cephfs delete pod ceph-csi-cephfs-provisioner-5f465c8b64-s5h7s
kubectl -n ceph-csi-cephfs delete pod ceph-csi-cephfs-provisioner-5f465c8b64-hz2zp
kubectl -n ceph-csi-cephfs delete pod ceph-csi-cephfs-provisioner-5f465c8b64-dt8m4I started on the troubleshooting steps at https://github.com/rook/rook/blob/master/Documentation/Troubleshooting/ceph-csi-common-issues.md
I tried provisioning again and logged into the cephfs-provisioner pod currently being attempted by the PVC provisioner.
kubectl -n ceph-csi-cephfs exec -ti pod/ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq -c csi-cephfsplugin -- basI verified with curl that I can connect to both ports on all five Ceph IPs from inside the provisioner pod.
I then copied the minimal ceph.conf configuration into the pod's /etc/ceph.conf file and ran ceph health, where I though I found the issue:
[root@ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq ceph]# vi ceph.conf
[root@ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq ceph]# ceph health
2023-10-18T23:38:04.587+0000 7fbea5583700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
2023-10-18T23:38:04.591+0000 7fbea4d82700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
2023-10-18T23:38:04.595+0000 7fbe9ffff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)From:
- https://github.com/ceph/ceph-csi/issues/2848
- https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/K7LDVS7Y5XQV7ILHC5WUWMXVJ5HX4HU3/
However, this was supposedly fixed in client 16.2.1 (https://docs.ceph.com/en/quincy/security/CVE-2021-20288/), and the pod is running Ceph client 16.2.5:
[root@ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq /]# ceph --version
ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)Continuing on, I copied the k8s-dev-cephfs keyring to the pod and it worked:
[root@ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq /]# ceph --id k8s-dev-cephfs health
HEALTH_WARN 1 pgs not deep-scrubbed in time
[root@ceph-csi-cephfs-provisioner-5f465c8b64-xnnfq /]# ceph --id k8s-dev-cephfs fs subvolumegroup ls cephfs
[
{
"name": "arctica-test-subvol-group"
},
...So as of now it appears that the contents of the secret csi-cephfs-secret are not being loaded in the provisioner pods. I believe this because the provisioning failed with the limited k8s-dev-cephfs user and the full admin account in the secret, and succeeded from the provisioner pod with the limited k8s-dev-cephfs user account.
I tried restarting the provisioner pods, thinking they might need to restart to load a new pod, but was not successful.
To be continued...
mbjones
We did not deploy Due to previous requirements that the CephFS CSI driver be granted admin access to the entire Ceph cluster: see https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md
However, this does not appear to be the case anymore, as limited CephFS access can be granted to K8s: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md#cephfs
My plan is to create a new k8s-dev-cephfs user in Ceph and update the
csi-cephfs-scstorage class, which is deployed, but not using valid credentials, and is not in use according tokubectl get pvc -A