search

DataONEorg / k8s-cluster

Documentation on the DataONE Kubernetes cluster
Apache License 2.0
2 stars 1 forks source link

Configure persistent volumes with ceph-csi cephfs or rbd #6

Closed gothub closed 2 years ago

gothub commented 3 years ago

As mentioned in https://github.com/DataONEorg/k8s-cluster/pull/2#issuecomment-854284130, now that CephFS has been setup on our k8s clusters, we can setup dynamic provisioning for persistent volumes.

This can be enabled on the dev cluster, as CephFS has been enabled for the control and worker nodes. Once this has been setup and tested, it can be setup for the production cluster (once the control node has been enabled for CephFS).

gothub commented 2 years ago

Regarding dynamic provisioning of k8s storage - it appears that the most up-to-date method to enable this is to use external (to the k8s distribution) storage provisioners that adhere to the Container Storage Interface.

This is important as there are legacy versions of these provisioners that might not be maintained, and it appears that the k8s documentation has not caught up, e.g. at https://kubernetes.io/docs/concepts/storage/storage-classes/. Installing the legacy version of the provisioner for CephFS didn't work, i.e. following the example at https://www.velotio.com/engineering-blog/kubernetes-storage-using-ceph.

Here is the location of the legacy CephFS provisioner that appears to no longer be maintained: Here is the legacy CephFS provisioner that appears to no longer be supported: https://github.com/kubernetes-retired/external-storage/tree/master/ceph/cephfs

The next step is to try the CSI CephFS found at https://github.com/ceph/ceph-csi/tree/devel/examples/cephfs

gothub commented 2 years ago

The Ceph cluster has been setup and made available for use by the k8s clusters as described here.

After attempting to setup ceph-csi rbd for k8s dev, I noticed that ceph-csi rbd doesn't support the access mode that is needed by k8s services - at least by the quality services, which is 'ReadWriteMany'. Here is a quick synopsis from the k8s docs:

A description of why ceph-csi rbd doesn't support ReadWriteMany is here.

Therefore, the next best option is to use statically provisioned ceph-csi cephfs persistent volumes, as described here.

@nickatnceas for this I need to setup the cephfs subvolume as detailed in the document mentioned above, for dev k8s initially, with commands such as:

ceph --name client.k8sdevrbd fs volume create k8sdevfs
ceph --name client.k8sdevrbd fs subvolumegroup create k8sdevfs k8sdevGroup -pool_layout k8sdev-pool-ec42-data
ceph --name client.k8sdevrbd fs subvolume create k8sdevfs k8sdevSubVolume k8sdevGroup --size=1073741824

These commands are supported in the currently installed ceph version (octapus version 15.2.14) accessible from k8s dev as described here.

Also relevant: https://docs.ceph.com/en/mimic/cephfs/file-layouts/#adding-a-data-pool-to-the-mds which states

Before you can use a pool with CephFS you have to add it to the Metadata Servers...

$ ceph fs add_data_pool cephfs cephfs_data_ssd
$ ceph fs ls  # Pool should now show up
...

@nickatnceas does that look reasonable to you?

nickatnceas commented 2 years ago

Hi @gothub , I deployed a subvol for the k8sdev CephFS. The username, secret, and path are all new, here's the info:

username: k8sdevsubvoluser subvolgroup: k8sdevsubvolgroup subvolume: k8sdevsubvol subvolume path: /volumes/k8sdevsubvolgroup/k8sdevsubvol/4b7cd044-4055-49c5-97b4-d1240d276856 secret: /etc/ceph/ceph.client.k8sdevsubvoluser.keyring (on docker-dev-ucsb-1 and docker-dev-ucsb-2)

Verifying that it works:

outin@docker-dev-ucsb-1:~$ sudo mount -t ceph 10.0.3.197:6789,10.0.3.207:6789,10.0.3.214:6789,10.0.3.222:6789,10.0.3.223:6789:/volumes/k8sdevsubvolgroup/k8sdevsubvol/4b7cd044-4055-49c5-97b4-d1240d276856 /mnt/k8sdevsubvol -o name=k8sdevsubvoluser,secretfile=/etc/ceph/k8sdevsubvoluser.secret
outin@docker-dev-ucsb-1:~$ df -Th /mnt/k8sdevsubvol
Filesystem                                                                                                                                                   Type  Size  Used Avail Use% Mounted on
10.0.3.197:6789,10.0.3.207:6789,10.0.3.214:6789,10.0.3.222:6789,10.0.3.223:6789:/volumes/k8sdevsubvolgroup/k8sdevsubvol/4b7cd044-4055-49c5-97b4-d1240d276856 ceph  889T     0  889T   0% /mnt/k8sdevsubvol
outin@docker-dev-ucsb-1:~$ cd /mnt/k8sdevsubvol/
outin@docker-dev-ucsb-1:/mnt/k8sdevsubvol$ sudo mkdir testdir
outin@docker-dev-ucsb-1:/mnt/k8sdevsubvol$ sudo chmod 777 testdir
outin@docker-dev-ucsb-1:/mnt/k8sdevsubvol$ echo hi > testdir/test.file
outin@docker-dev-ucsb-1:/mnt/k8sdevsubvol$ cat testdir/test.file
hi
outin@docker-dev-ucsb-1:/mnt/k8sdevsubvol$ sudo rm -rf testdir

The commands I used to create the subvol are at https://github.nceas.ucsb.edu/NCEAS/Computing/blob/master/cephfs.md#commands-used-to-create-cephfs-subvolumes-on-the-dataone-ceph-cluter-ceph-15

I'll setup prod next.

nickatnceas commented 2 years ago

Production is now ready and tested.

username: k8ssubvoluser subvolgroup: k8ssubvolgroup subvolume: k8ssubvol subvolume path: /volumes/k8ssubvolgroup/k8ssubvol/af348873-2be8-4a99-b1c1-ed2c80fe098b secret: /etc/ceph/ceph.client.k8ssubvoluser.keyring (on docker-ucsb-4, 5, 6, and 7)

root@docker-ucsb-4:~# mount -t ceph 10.0.3.197:6789,10.0.3.207:6789,10.0.3.214:6789,10.0.3.222:6789,10.0.3.223:6789:/volumes/k8ssubvolgroup/k8ssubvol/af348873-2be8-4a99-b1c1-ed2c80fe098b /mnt/k8ssubvol -o name=
k8ssubvoluser,secretfile=/etc/ceph/k8ssubvoluser.secret
root@docker-ucsb-4:/mnt/k8ssubvol# df -Th /mnt/k8ssubvol
Filesystem                                                                                                                                             Type  Size  Used Avail Use% Mounted on
10.0.3.197:6789,10.0.3.207:6789,10.0.3.214:6789,10.0.3.222:6789,10.0.3.223:6789:/volumes/k8ssubvolgroup/k8ssubvol/af348873-2be8-4a99-b1c1-ed2c80fe098b ceph  888T     0  888T   0% /mnt/k8ssubvol
root@docker-ucsb-4:~# cd /mnt/k8ssubvol/
root@docker-ucsb-4:/mnt/k8ssubvol# mkdir testdir
root@docker-ucsb-4:/mnt/k8ssubvol# echo hi > testdir/test.file
root@docker-ucsb-4:/mnt/k8ssubvol# cat testdir/test.file
hi
root@docker-ucsb-4:/mnt/k8ssubvol# rm -rf testdir
gothub commented 2 years ago

ceph-csi for cephfs has been configured on dev k8s and metadig-engine is currently running using a persistent volume (PV) provisioned with ceph-csi. This PV uses the subvolume described above. The steps to install and configure ceph-csi the PV and persistent volume claim will be described https://github.com/DataONEorg/k8s-cluster/blob/develop/storage/Ceph/Ceph-CSI.md (I have a bit more work to do on the docs).

Although this test was done by installing and configuring ceph-csi by manually editing k8s manifest files (.yaml), the goal is to perform the install, config and updates using Helm.

gothub commented 2 years ago

ceph-csi cephfs plugin is now installed using Helm. Instructions are here