Add Rook addon

[alexkreidler]

Rook (https://rook.io/) is a platform for adding CRDs and Operators to Kubernetes to provision various types of Cloud-Native distributed storage systems.

It would be awesome if we could enable it in microk8s with a simple script (microk8s.enable rook).

I think a lot of people would find it very interesting to experiment with various storage systems and DBs on their local machine to then test them in other environments later. I also think that microk8s has a unique position of offering new and useful k8s tools like istio, jaeger, etc so people can discover new things.

For implementation, it seems like

1. rbd (RADOS block device), a kernel module needed for Rook has support from kernel >~ 3.10 up, but for using a CephFS storage backend, you need >4.17. All we'd need to do is sudo modprobe rbd
2. Rook also needs a permissive PodSecurityPolicy to run properly. This guide seems to have the best information on how to do that with microk8s, we would just need to add these scripts to the core. Section 8 of the guide shows then configuring a relaxed PSP for 1 namespace, which we could do for the Rook namespace we create.

microk8s is without a doubt the best local k8s environment of them all, so thanks for all the hard work!

I'd love any and all feedback on this and I'd be happy to start working on it if I got a sense of if anyone else would find it useful!

[balchua]

@alexkreidler i love to see rook as an addon. This can be very useful to have especially when running microk8s as a cluster, since hostPath provisioner is going to be unusable on a multi node microk8s. Getting rook + cephfs up and running with an easy command like microk8s.enable rook will be great!

[ktsakalozos]

@alexkreidler I am +1 too! How would the UX look like? Are you thinking of a rook addon that takes as parameter whatever needs to be setup eg microk8s.enable rook:cephfs?

[trulede]

I would like to use MicroK8s in a small cluster, the only thing preventing me in doing that is a lack of cluster wide storage. If it can be done with Rook, that would be great. It would be nice, for simplicity sake, if all the participating nodes would have their hostPaths wrapped up into a cluster wide storage, or similar ... at least as a start point.

[balchua]

The simplest i can think of is an nfs provisioner. But making nfs HA is not something i have done. 😁

[trulede]

I took a crack at getting Rook working today, on MicroK8s, with CephFS. I was not successful, not sure why. One takeaway was that I need to add setting "--allow-privileged" to kube-apiserver. Otherwise, its likely that all the storage types are interesting; Block, Object and CephFS.

Despite it not working, Rook was very nice ... there is just a lot of stuff.

[ktsakalozos]

I need to add setting "--allow-privileged"

You know how to do that, right?

[trulede]

Yep. I guess that the MicoK8s side of things was OK, and that somewhere the Rook/Ceph was misconfigured - perhaps only for MicroK8s. I will try again in the next days.

Perhaps I should ask, has anyone been able to get a Rook Ceph stack running on MicroK8s?

At least I was hoping/trying to get some storage from the local disk (microk8s.io/hostpath) available via CephFS, which would then be available "cluster-wide". But Ceph never created the volume, so something was not wired up correctly.

I was following this guide : https://rook.io/docs/rook/v1.2/ceph-quickstart.html and files in this folder (mostly) : cluster/examples/kubernetes/ceph/

[balchua]

But all components are running? Also wanted to try and install rook.

[trulede]

All containers were running, so it looked OK, however when I would create a volume ... no event was picked up by the Ceph Operator (which should then actually create the volume). Some others had similar problems on other platforms, so I guess a bit of frailty, or misconfiguration on my part.

When I try again, I will pay a bit more attention ;-)

[strigona-worksight]

Rook is on my list of things to try out and I came across this thread a little while ago: https://github.com/rook/rook/issues/4006 perhaps there's something useful in there?

[trulede]

Rook is no longer supporting directories, or are in the process of not supporting that, which is I guess what was preventing me from getting a Cluster running (seemed that way). That leaves us with Block Devices ... and perhaps PVC's.

Block devices should "just work", however that would be a host configuration (I think). A PVC would be an interesting way, I did try that ... with the microk8s-hostpath storageclass ... but the hostpath-provisioner POD went to 100% CPU (a microk8s.stop/start fixed that). Will try again ...

[balchua]

@trulede was just wondering what you meant by "no longer supporting directories".

I also tried my luck with rook. The farthest i was able to reach is to have the CephCluster components such as mon, mgr etc all running. Then i started creating the StorageClass for cephfs and pvc. This is where i got stuck... The pvc is stuck in Pending status. 😭.

[trulede]

You need to make sure you have some "osd" pods running. They should come into existence based on your cluster.yaml file (applied after common.yaml and operator.yaml). I edit mine for a single node (default is 3), which is also necessary if running only one node.

Now, that is where I'm stuck. Seems I need a block device, so I will try with a loopback device next ... and hopefully it will get picked up and the "osd" pod will be created. After that the rest should work.

It might also be possible to hook into a PVC rather than a Block device ... but easy things first :-)

[trulede]

No luck with a loopback:

 k -n rook-ceph logs rook-ceph-osd-prepare-ubuntu-bionic-klcss | grep loop
2020-03-10 06:46:57.436531 I | exec: Running command: lsblk /dev/loop8 --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME,NAME
2020-03-10 06:46:57.438018 W | inventory: skipping device "loop8" because 'lsblk' failed. unsupported diskType loop

Next is to try to add a disk to the VM I'm using ...

[trulede]

No luck with a PVC either :-)

The problem I had was that Rook wants a Storageclass/PVC with Block storage. AFAIKS the microk8s-local storage class is Filesystem only. It makes sense.

Is there a a raw block storage provider for MicroK8s? My feeling is that the only way this will work is with some kind of block device (probably adding a disk). That I will try in the next days.

[trulede]

So, to use Rook as a storage provider, you need a Block device as the basis for the storage. Depending on how you are running MicroK8s that can mean different things. For me, using Virtual Box, I added an additional SCSI VDI to my VM, and then Rook was able to pick that up.

😄

git clone https://github.com/rook/rook.git
cd rook/cluster/examples/kubernetes/ceph
k create -f common.yaml
k create -f operator.yaml
k -n rook-ceph get pods
vim cluster.yaml
    edit mon: count: 1
    edit storage: useAllDevices: true
k create -f cluster.yaml

if all is good you will eventually have an OSD Pod running. Check the status:

k create -f toolbox.yaml
k -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
# ceph status
  cluster:
    id:     d4e9dcbc-ac5b-47d1-82f3-2915924366c5
    health: HEALTH_WARN
            OSD count 1 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum a (age 11m)
    mgr: a(active, since 10m)
    osd: 1 osds: 1 up (since 9m), 1 in (since 9m)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   1.0 GiB used, 13 GiB / 14 GiB avail
    pgs:

Next, determine how you want to provision that storage: Block, Object or Shared filesystem.

[balchua]

This is cool! Have u tried creating a pvc with cephfs?

[trulede]

Yes, it took some effort, but I have it working. I needed to adjust the cluster.xml setting network: hostNetwork: true. It was a monumental PITA to get this working 😄 and I assume its working .... because there are so many layers where things can go wrong ...

The config files I used are here:

https://github.com/trulede/mk8s_rook.git

Somebody with more computers than me, and perhaps more sense (or less), might also want to try a cluster setup.

[balchua]

@trulede i manage to make it cephfs pvc work on a multi node microk8s. I was running it in DigitalOcean, and i need to add a block device to my Droplets.

I am still figuring out how to run this on my laptop. I only have one disk attached to my laptop. I agree with you that this is tough to setup. If we are going to have rook-ceph as addon, we need to have a good instruction on how to add a device, this is where it gets tricky (atleast for me).

[trulede]

@balchua That's great! And very interesting ... MicroK8s with Rook would be a very viable solution for building clusters.

I use vagrant/virtual_box, so adding a block device is actually fairly easy to do. But it will be different for everyone (I don't know what a droplet is 😊). If someone is going to build a cluster, with Rook/CepfFS providing some shared in cluster storage, they will probably manage.

However, if it is somehow possible to overlay a Block device on the microk8s-hostpath provisioner, then it would be possible to create a Block Device PVC which could be consumed by Rook. That may not be a good idea for reliable storage (I don't know), but it could get people started.

[balchua]

I also tried rook's nfs on top of microk8s hostpath storage. And it is far easier than ceph.

Although i pinned down the hostpath provisioner to a node (if you have a multi node cluster).

My only gripe with rook + ceph, is that it doesn't provide a good alert when there is no block device available for the osd to use.

[trulede]

Rook as an addon, with a default Rook NFS Service running from a host, would be OK. Perhaps create the storage classes too, which enables dynamic volume provisioning.

Having NFS as a service, like Ingress (and others), which can be enabled in MicroK8s would certainly be a handy thing. 👍

$ microk8s.enable nfs and create the required Pods in an nfs namespace, perhaps? It would certainly be nice to then have the storage enabled, and a storage class there, as well as any ingress needed to access the nfs from a host machine. Very positive outcome in any case. I will try the NFS later today or tomorrow.

[VestigeJ]

I'm trying to work through this right now, I've got a nearly identical 6 node physical cluster (thank you old mini desktops). I've also got dedicated drives for OSD, although I'm thinking I should've installed the OS's to USBs instead of one of the drives. The computers aren't much they're quad core i5's with 2 500GB WD SSDs installed.

I'm a one person team trying to learn in a barely dev (almost production) environment so wish me luck...

About to try to make the modifications some of you have mentioned regarding the pod policies that are preventing the operator from starting. Thank you all for the links, hopefully we can continue to drive this forward. I've got a decent test environment if any of you have ideas for me to try and document.

[balchua]

@VestigeJ i've got my stuffs here. All rook related manifests are in the folder rook.

[stephen304]

I'm having a lot of trouble trying to follow the rook docs to get block storage:

https://rook.io/docs/rook/v1.5/ceph-block.html

Up until that point has been pretty smooth, except /var/lib/kubelet/plugins_registry and pods not existing, so I created those on all my nodes.

Then when I try to consume the storage, the pvs create and get bound to the pods, but it fails to mount. The pods show:

  Normal   Scheduled    60m                   default-scheduler  Successfully assigned stephen304/wordpress-7b989dbf57-hrl9f to nebula-1
  Warning  FailedMount  29m (x3 over 38m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[default-token-qhw62 wordpress-persistent-storage]: timed out waiting for the condition
  Warning  FailedMount  13m (x24 over 59m)    kubelet            MountVolume.MountDevice failed for volume "pvc-d9cc6051-db3d-4321-9b13-544d59319acd" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers
  Warning  FailedMount  4m37s (x20 over 58m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[wordpress-persistent-storage default-token-qhw62]: timed out waiting for the condition

It seems to be related to csi/rbd plugin, but I'm not sure how to diagnose.

[balchua]

@stephen304 its been a long time since i tried it with microk8s. But i remember i do have to change the rook-ceph-operator args to point to microk8s kubelet args: ["ceph", "operator", "--csi-kubelet-dir-path", "/var/snap/microk8s/common/var/lib/kubelet"]

I have kept my rook manifests here https://github.com/balchua/do-microk8s/tree/master/rook in case you find some difference. I was able to make rook work using these manifests. I ran these on DigitalOcean though.

Some few instructions https://github.com/balchua/do-microk8s#digitalocean-attached-volume

[stephen304]

@balasu Oh thanks! I didn't realize that /var/lib/kubelet wasn't the right one. It looks like that flag is replaced with ROOK_CSI_KUBELET_DIR_PATH, that was confusing me a bit. It seems that was the key to getting it to work :) Unfortunately my csi-rbdplugin-provisioner and csi-cephfsplugin-provisioner pods are still in CrashLoopBackoff, I'm hoping that's just something I messed up along the way.

[stephen304]

I got around to re-setting up my cluster from scratch. I didn't touch anything other than adding ROOK_CSI_KUBELET_DIR_PATH, but it seems the 2 deployments csi-rbdplugin-provisioner and csi-cephfsplugin-provisioner are still having issues. My manifests are located here but for brevity, the only difference between my manifests and the 1.5.3 ceph examples referenced in the getting started docs is this:

--- a/cluster/rook/operator.yaml
+++ b/cluster/rook/operator.yaml
@@ -65,7 +65,7 @@ data:
   # CSI_RBD_PLUGIN_UPDATE_STRATEGY: "OnDelete"

   # kubelet directory path, if kubelet configured to use other than /var/lib/kubelet path.
-  ROOK_CSI_KUBELET_DIR_PATH: "/var/snap/microk8s/common/var/lib/kubelet"
+  # ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet"

   # Labels to add to the CSI CephFS Deployments and DaemonSets Pods.
   # ROOK_CSI_CEPHFS_POD_LABELS: "key1=value1,key2=value2"
@@ -418,7 +418,7 @@ spec:
         # Whether to start the discovery daemon to watch for raw storage devices on nodes in the cluster.
         # This daemon does not need to run if you are only going to create your OSDs based on StorageClassDeviceSets with PVCs.
         - name: ROOK_ENABLE_DISCOVERY_DAEMON
-          value: "true"
+          value: "false"

         # Time to wait until the node controller will move Rook pods to other
         # nodes after detecting an unreachable node.

Inside both the cephfsplugin-provisioner and rbdplugin-provisioner deployment and for both pods of each deployment, the csi-attacher, csi-provisioner, csi-snapshotter, and csi-resizer are in crashbackoffloop.

The logs seem to mostly look like this:

┌─────────────────────────────────────────────────── Logs(rook-ceph/csi-cephfsplugin-provisioner-7dc78747bf-4cdtq:csi-attacher)[all] ────────────────────────────────────────────────────┐
│                                                            Autoscroll:On     FullScreen:Off     Timestamps:Off     Wrap:Off                                                            │
│                                                                                                                                                                                        │
│ I1210 20:09:00.407479       1 main.go:91] Version: v3.0.0                                                                                                                              │
│ I1210 20:09:00.519862       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock                                                                                        │
│ I1210 20:09:00.520639       1 common.go:111] Probing CSI driver for readiness                                                                                                          │
│ W1210 20:09:00.524271       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.                                                        │
│ I1210 20:09:00.662324       1 leaderelection.go:243] attempting to acquire leader lease  rook-ceph/external-attacher-leader-rook-ceph-cephfs-csi-ceph-com...                           │
│ E1210 20:09:30.156377       1 leaderelection.go:357] Failed to update lock: Operation cannot be fulfilled on leases.coordination.k8s.io "external-attacher-leader-rook-ceph-cephfs-csi │
│ I1210 20:10:56.188019       1 leaderelection.go:253] successfully acquired lease rook-ceph/external-attacher-leader-rook-ceph-cephfs-csi-ceph-com                                      │
│ I1210 20:10:56.188223       1 controller.go:121] Starting CSI attacher                                                                                                                 │
│ E1210 20:11:06.188548       1 leaderelection.go:357] Failed to update lock: Put "https://10.152.183.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/external-attacher-le │
│ I1210 20:11:06.188741       1 leaderelection.go:278] failed to renew lease rook-ceph/external-attacher-leader-rook-ceph-cephfs-csi-ceph-com: timed out waiting for the condition       │
│ F1210 20:11:06.188797       1 leader_election.go:169] stopped leading                                                                                                                  │
│ E1210 20:11:06.190155       1 event.go:273] Unable to write event: 'can't create an event with namespace 'default' in namespace 'rook-ceph'' (may retry after sleeping)                │
│ I1210 20:11:06.190192       1 controller.go:143] Shutting CSI attacher                                                                                                                 │
│ stream closed  

[balchua]

@stephen304 a stupid question here, do you have a different block store in your system?

[stephen304]

@balchua Not that I'm aware of, I pretty much just installed microk8s on 3 fresh vms, enabled dns and ingress, then applied the rook manifests. It seems every once in a while I can get storage provisioned and pvs created, but I assume since the aforementioned pods are failing most of the time I can't.

[balchua]

sorry i mean block device. Im guessing you have, as you mentioned you do occasionally get some PVs created. Im not entirely familiar with Rook. Although i have to admit that its not that easy to setup. I recently been experimenting with longhorn. It is far easier to get it running and you can easily start playing with it. But i have no idea about its performance.

[miro-balaz]

I was able to install it on microk8s v1.21.1, with little work, only change from default rook-ceph instalaton is path to kubelet (inside operator.yaml)

[woutersamaey]

Any updates on this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sachinkumarsingh092]

Hey all, the rook-ceph addon will be available with microk8s v1.28.