Cannot reset glsuterfs topology

[bend]

I want to reset the glusterfs topology because one of my server changed IP.

So I tried to delete everything related to glusterfs:

kubectl delete -n gluster svc,deployments,daemonset,pods,pv,pvc --all

I also deleted and recreated the volumes on all the nodes I delete the /etc/glusterfs directory on all the nodes I've edited the topology to match the new one (one IP changed)

I then ran:

./gk-deploy -n gluster -w 900 -g -y topology.json

But the still get the error:

Using Kubernetes CLI.                                                                                                                                       
Using namespace "gluster".                                                                                                                                  
Checking for pre-existing resources...                                                                                                                      
  GlusterFS pods ... found.                                                                                                                                 
  deploy-heketi pod ... found.                                                                                                                              
  heketi pod ... not found.                                                                                                                                 
  gluster-s3 pod ... not found.                                                                                                                             
Creating initial resources ... Error from server (AlreadyExists): error when creating "/home/user/k8s/gluster-kubernetes/deploy/kube-templates/heketi-service
-account.yaml": serviceaccounts "heketi-service-account" already exists                                                                                     
Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io "heketi-sa-view" already exists                                            
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view not labeled                                                                                     
OK                                                                                                                                                          
Found node vps01 on cluster d1590f4ee8fcfb6202446c8b63b95908                                                                                                
Found device /dev/loop0                                                                                                                                     
Found node vps02 on cluster d1590f4ee8fcfb6202446c8b63b95908                                                                                                
Found device /dev/loop0                                                                                                                                     
Found node vps04 on cluster d1590f4ee8fcfb6202446c8b63b95908                                                                                                
Found device /dev/loop0                                                                                                                                     
heketi topology loaded.                                                                                                                                     
Error: Unable to execute command on glusterfs-wb4cl: volume create: heketidbstorage: failed: Host 5.196.XX.XXX is not in 'Peer in Cluster' state            
command terminated with exit code 255                                                                                                                       
Failed on setup openshift heketi storage                                                                                                                    
This may indicate that the storage must be wiped and the GlusterFS nodes must be reset. 

The IP displayed in the error message does not match the one in my topology.json file.

Is there something I forgot to delete in order to completely reset the glusterfs cluster ? I don't care about the data as I'm still trying to setup a cluster.

[SaravanaStorageNetwork]

As per log, gluster pods still found. You need to delete all pods first.

You can try kubectl get all and delete all resources.

Also, by passing --abort you can delete existing resources.

[bend]

Yes it works now, however I have now this error: mount: /var/lib/kubelet/pods/ae815ad6-ac5a-11e8-bbc2-fa163eec9a70/volumes/kubernetes.io~glusterfs/heketi-storage: unknown filesystem type 'glusterfs'.

Do I need to install glusterfs-fuse on the host machine ? Is there any package required to be installed on the nodes host machines ?

[SaravanaStorageNetwork]

Do I need to install glusterfs-fuse on the host machine ?

GlusterFS client version installed on nodes should be as close as possible to the version of the server.

Please check https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md

[bend]

Ok It worked. Thank you

[SaravanaStorageNetwork]

@bend Glad it helped - If you think there is any need to improve anything specific in documentation, let us know. Also, feel free to send Pull requests :)