Unable to provision Volume using Heketi with 1 only node on GlusterFS

devilmind81 commented 5 years ago

Hi all, I have exattly the same issue described here #454 but all suggestion is not fixing my issue. I have been setup glusterfs on one node in a dev environment and I'm trying to add a StorageClass in the openshift in order to manage the volume creation on it.

This is my StorageClass yaml file content:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.10.10.10:8080"
  restuser: "admin"
  restuserkey: "admin"
  **volumetype: "replicate:1"**

and this is my pvc yaml file content:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: heketi-pvc
 annotations:
   volume.beta.kubernetes.io/storage-class: heketi
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
     storage: 4Gi

but when I create a pvc I'm obtaining:

Name:          heketi-pvc
Namespace:     test
StorageClass:  heketi
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class=heketi
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
Finalizers:    []
Capacity:
Access Modes:
Events:
  Type     Reason              Age   From                         Message
  ----     ------              ----  ----                         -------
  Warning  ProvisioningFailed  3s    persistentvolume-controller  Failed to provision volume with StorageClass "heketi": create volume error: error creating volume Failed to allocate new volume: No space

in the heketi logs I see:


Nov 27 17:39:58 ocpglusterfs-0 heketi[14686]: [heketi] ERROR 2018/11/27 17:39:58 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:37: Minimum brick size limit reached.  Out of space.
Nov 27 17:39:58 ocpglusterfs-0 heketi[14686]: [heketi] ERROR 2018/11/27 17:39:58 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:1167: Create Volume Build Failed: No space

I see that the follow topology has something that sounds no good for me, the Free (Gib):

Cluster Id: 913b61c4b6429e56550cfb4958f4a63c

    File:  true
    Block: true

    Volumes:

    Nodes:

        Node Id: b374dccf5b3ae3ea2576a9705e12cada
        State: online
        Cluster Id: 913b61c4b6429e56550cfb4958f4a63c
        Zone: 1
        Management Hostnames: 10.10.10.10
        Storage Hostnames: 10.10.10.10
        Devices:
                Id:3dd5b7772a685f538696c56359665b6f   Name:/dev/sdb            State:online    Size (GiB):49      Used (GiB):49      Free (GiB):0

The glusterfs and heketi were installed from scratch, why hasn't available space?

Can someone help me here? I'm stuck and I don't know how to proceed. Thanks in advance.

PS: I read the documentation at link: https://kubernetes.io/docs/concepts/storage/storage-classes/#glusterfs and I have tried also with **volumetype: "none"** but the same issue occurs.

nixpanic commented 5 years ago

You're very close! Please try with volumetype: none in your StorageClass.

devilmind81 commented 5 years ago

already tried, same issue. other suggestion? why I haven't GB available if the topology is just created?

phlogistonjohn commented 5 years ago

@devilmind81 since the topology shows zero free space but no volumes it may be that you have volumes that were partially created. These are called pending volumes in heketi. If the operation associated with the pending volume never completed it can occupy space be be invisible to the listing.

To determine if this is the case you can try either of the below things:

heketi-cli server operations info and look at the count of stale operations. If it's non-zero you have stale operations in your db
If you have an older version of heketi that does not have the above command or want more detail you can try heketi-cli db dump and look through the json for the "pendingoperations" key. If there's anything under that key that means you have pending ops in your db.

If this is the case I can try to help remove the pending operations from heketi and any underlying components from the storage nodes. But let's confirm that this is the issue first.

devilmind81 commented 5 years ago

@phlogistonjohn thank you for you reply. This is the output of the db dump command:

{"clusterentries":{"913b61c4b6429e56550cfb4958f4a63c":{"Info":{"id":"913b61c4b6429e56550cfb4958f4a63c","nodes":["b374dccf5b3ae3ea2576a9705e12cada"],"volumes":[],"block":true,"file":true,"blockvolumes":[]}}},"volumeentries":{},"brickentries":{},"nodeentries":{"b374dccf5b3ae3ea2576a9705e12cada":{"State":"online","Info":{"zone":1,"hostnames":{"manage":["10.10.10.10"],"storage":["10.10.10.10"]},"cluster":"913b61c4b6429e56550cfb4958f4a63c","id":"b374dccf5b3ae3ea2576a9705e12cada"},"Devices":["3dd5b7772a685f538696c56359665b6f"]}},"deviceentries":{"3dd5b7772a685f538696c56359665b6f":{"State":"online","Info":{"name":"/dev/sdb","storage":{"total":52293632,"free":610304,"used":51683328},"id":"3dd5b7772a685f538696c56359665b6f"},"Bricks":[],"NodeId":"b374dccf5b3ae3ea2576a9705e12cada","ExtentSize":4096}},"blockvolumeentries":{},"dbattributeentries":{"DB_CLUSTER_HAS_FILE_BLOCK_FLAG":{"Key":"DB_CLUSTER_HAS_FILE_BLOCK_FLAG","Value":"yes"},"DB_GENERATION_ID":{"Key":"DB_GENERATION_ID","Value":"54aa1c8732a814875d74030199f56428"},"DB_HAS_PENDING_OPS_BUCKET":{"Key":"DB_HAS_PENDING_OPS_BUCKET","Value":"yes"}},"pendingoperations":{}}

If i delete the db, reboot the vm and create a new cluster I have noticed that after creation I have all free Gib free but after some min I have no free space available again.....why?

devilmind81 commented 5 years ago

I ve fixed the issue. This is what I have done:

I have created a LVM dedicated on the my disk /dev/sdb1 (50 Gb):

vgcreate gluster_group /dev/sdb1
lvcreate -l 100 -n gluster gluster_group 
lvextend -L50000 /dev/gluster_group/gluster

Afer that I have recreated the heketi for that new topology:

{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "10.10.10.10"
                            ],
                            "storage": [
                                "10.10.10.10"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        {
                            "name": "/dev/gluster_group/gluster",
                            "destroydata": false
                        }
                    ]
                }
                    ]
                }
            ]
        }
    ]
}

After that I have all GB free in heketi and the volumes are able to create successfully from Openshift.

Thanks a lot to all

gluster / gluster-kubernetes

Unable to provision Volume using Heketi with 1 only node on GlusterFS #540