How to stop/restart heketi/glusterFS pods without losing data

LazyTuba commented 5 years ago

I find docs on bringing up brand new gluster filesystem. I am looking for guidance on performing routine maintenance without having to wipe block devices and starting over

If I need to briefly shut down the heketi/gluster processes, how can I do that and then bring them back up without losing data on the volumes? (I have an immediate need for this )
How do I upgrade Gluster without losing data
If I reset the K8S cluster running GLuster pods – i.e., basically wiping K8S off the node - can I regain access to the pre-reset GLuster storage once I bring K8S back up?

phlogistonjohn commented 5 years ago

Your best bet is to look at the Red Hat OCS documentation which pretty much uses this same stack: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/deployment_guide/#part-Upgrade

Generally the rule of thumb is to check that your gluster nodes do not have any outstanding heals. Use the gluster volume heal VOLNAME info command to check for this. If you have no heals you can stop the storage nodes. In addition, heketi has a feature called administrative modes which you can use to prevent heketi from making any additional changes to the system. Run: heketi-cli server mode set read-only before stopping any pods.

I think that the only way to accompish the latter would be with an external gluster and static provisioning. This is not really the topic covered by gluster-kubernetes and heketi and so I can't speak much to that.

LazyTuba commented 5 years ago

phlogistonjohn, Thank you for the pointers.

gluster / gluster-kubernetes

How to stop/restart heketi/glusterFS pods without losing data #566