kadalu / kadalu

A lightweight Persistent storage solution for Kubernetes / OpenShift / Nomad using GlusterFS in background. More information at https://kadalu.tech
https://docs.kadalu.tech/k8s-storage/devel/quick-start/
Other
713 stars 99 forks source link

[Bug]: ResourceExhausted desc = No Hosting Volumes available, add more storage #1061

Open chuegel opened 6 months ago

chuegel commented 6 months ago

Describe the bug Running Kadalu 1.2.0 with Replica3.

apiVersion: kadalu-operator.storage/v1alpha1
kind: KadaluStorage
metadata:
  name: kadalu-pool-1
  namespace: kadalu
spec:
  type: Replica3
  storage:
    - node: k8s-04-mgmt-net
      device: /dev/vg1/k8s
    - node: k8s-05-mgmt-net
      device: /dev/vg1/k8s
    - node: k8s-06-mgmt-net
      device: /dev/vg1/k8s

When provisioning a volume through a storage class the pvc get stucked in:

failed to provision volume with StorageClass "kadalu.kadalu-pool-1": rpc error: code = ResourceExhausted desc = No Hosting Volumes available, add more storage

There is plenty of space left:

df -h | grep /dev/mapper/vg1-k8s
/dev/mapper/vg1-k8s   200G  2,4G  198G   2% /k8s

To Reproduce Steps to reproduce the behavior:

  1. install 1.2.0 with Replica3
  2. provision a volume
  3. watch the pvc's getting stucked with the error above

Expected behavior Kadalu should be able to provision the volumes without the error described above

Actual behavior A clear and concise description of what actually happened.

Environment:

kubectl-kadalu plugin: 1.2.0
kadalu pod(s) versions
pod/kadalu-csi-nodeplugin-mx5kk: 1.2.0
pod/kadalu-csi-nodeplugin-x9tnj: 1.2.0
pod/kadalu-csi-nodeplugin-zw5mc: 1.2.0
pod/kadalu-csi-provisioner-0: 1.2.0
pod/operator-68856c9949-zk97k: 1.2.0
pod/server-kadalu-pool-1-0-0: 1.2.0
pod/server-kadalu-pool-1-1-0: 1.2.0
pod/server-kadalu-pool-1-2-0: 1.2.0

Screenshots or Logs If applicable, add any screenshots to help explain your problem. Try capturing relevant logs, post them to http://gist.github.com/ and post the links in the issue.

Additional context Add any other context about the problem here.

chuegel commented 6 months ago

I guess the storage is not calculated correctly:

kubectl kadalu storage-list --status

Name             Type        Utilization            Pvs Count      Min PV Size      Avg PV Size      Max PV Size
kadalu-pool-1    Replica3    199 Gi/199 Gi (99%)           20             2 Gi             9 Gi            20 Gi