Allowing local-storage as the storage option instead of LVM only?

[virtuman]

In most of our DEV clusters we try to give as much space to elasticsearch stack as possible to allow for better logging during the development process.

Here's a small list of reasons why I think this is a very viable option for both DEV and PROD use-cases:

1. Curator can be configured to delete older indexes when certain disk-space-utilization is reached, ie. 80% disk full - make at least nGB or y% by deleting older log indexes.
2. We spin up hundreds of environments during the development and having disks chopped up into LVMs - renders them completely underutilized during the development and while we have tens of terrabytes readily available - we end up not having enough space to deploy other review environments.
3. In our PROD - all volumes grow exponentially and we add disks for shared utilization as needs arise. Ie. Mongodb for stale/archived data is sharing disks with some of the services that are more "disk-hungry" to balance the utilization of hardware

those are primarily main reasons that make clustered services + local-storage use case a lot more desirable.

I understand that expandable LVM-based volumes is a features that is currently in progress, but still, it doesn't replace the need and ease of local provisioner for many use-cases, and should be up to the user to choose to use or not to use local-storage (local provisioner) over or along with LVM-managed volumes.

What are you thoughts on this?

Besides, it was mentioned in #1 from the very beginning of the planning, so you already considered the need for it from the very first discussions?

[NicolasT]

If I understand correctly, you'd like to use the local-storage provisioner as deployed (optionally) by Kubespray, and let this be used by the services deployed by MetalK8s instead of using LVM?

This should be possible: anything Kubespray does can be configured/set up in your inventory. Then the question becomes how to disable LVM provisioning and setup of the related StorageClass. This may be possible already by skipping tags during playbook executions. Otherwise, we may need to add a configuration flag to disable the related steps. PR welcome.

However, it is a goal of MetalK8s to 'work out of the box' and result in a production-grade cluster, so the defaults will remain as-is.

Maybe another alternative you could consider is to use MetalK8s' support to use pre-existing LVM VGs (this needs to be documented though), which would allow to share VGs across multiple installations (though, granted, you may run into LV naming conflicts... not sure how to fix that :thinking: ), and could even use thin/sparse provisioning of LVM.

[virtuman]

Actually, everything is already almost working correctly.

Current behavior is to to the follow:

1. provision LVMs (if any defined)
2. deploy kubespray cluster
3. deploy services
4. services expect default storage class
5. PROBLEM IS HERE: if no default storage exists (kubespray doesn't actually mark local-storage as a default StorageClass, or did I miss the option for it?) - services get stuck in deploying state as they try to provision PVC from no-default storageclass. I'm actually unable to figure out why PVCs wouldn't provision properly after i add "is-default" annotation to the local-storage storageclass.

Currently, to solve the "stuck" in deployment services (elastic + prometheus):

1. When I update local-storage after metal-k8s provisioning is completed - it doesn't actually pick up the newly assigned default storageclass and just continues to hang. So I have to continue on to the next steps:
2. Need to delete services with all resources that depend on persistent storage
3. Re-run services.yml playbook after which everything is great and seems to be working correctly.

Solution for solving this: There's already a variable called metalk8s_default_storageclass , which should be processed independently of all other LVM / VG tasks - prior to deploying services.yml. Currently it is processed at the time of storage-class creation here: https://github.com/scality/metalk8s/blob/5330d94115f6517f807743ae9160cdf830b019f8/roles/kube_lvm_storageclass/templates/local-storageclass.yml.j2#L6 , but should be a separate task, probably similar to this:

1. reset "storageclass.kubernetes.io/is-default-class":"false" on all existing StorageClasses (just in case)
2. and then mark desired StorageClass as default with something like kubectl patch storageclass {{ metalk8s_default_storageclass }} -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
3. OPTIONAL (added value) - would probably be better to completely delete local-storage existing storageclass, and re-create local-storage using your already existing storageclass template, where reclaimPolicy is defined as RETAIN, (otherwise, default StorageClass that kubespray deploys is with default reclaimPolicy, which is DELETE)