Distributed storage with PV provisioning

[edsealing]

Currently, we are using NFS for the PV provisioning. This has a few disavdantages:

• It only stores the data on a single host (no redundancy/single point of failure)
• Doesn't take advantage of the bulk storage across the cluster.
• No elasticity

Long-term we need to look into moving to a distributed storage system for persistent data. Ceph and GlusterFS are the current leading stable technologies within this space and both have support for Kubernetes. The solution should meet a few requirements:

1. Be fully containerized (nothing required of the host system)
2. Be able easily integrated into the node installation system so that the available storage seamlessly expands with the addition of new nodes.
3. Support auto-provisioning within kubernetes.

Please reply to this issue with potential solutions.

[edsealing]

Example of a GlusterFS deployment in kubernetes using containers: https://blog.lwolf.org/post/how-i-deployed-glusterfs-cluster-to-kubernetes/

GlusterFS 4.0 came out in March 2018. It's a Short Term Maintenance (STM) release. Version 4.1 is due out in June 2018 and will be a Long Term Maintenance (LTM) release. Some of the noted changes are improved kubernetes integration.

Guides for setting-up GlusterFS within K8S: https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md

Guide for setting-up Heketi: https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md