Unable to create Volume due to ResourceExhausted, but the Resources aren't exhausted.

[dmonoquant]

Describe the bug When trying to create an 800GB volume, I get resource exhausted, other volumes work without issue, and there is enough storage available.

To Reproduce

git clone https://github.com/spdk/spdk
cd spdk
sudo modprobe nvme_tcp
echo "nvme_tcp" | sudo tee -a /etc/modules
sudo apt-get install xfsprogs
sudo sed -i '/^GRUB_CMDLINE_LINUX_DEFAULT=/s/"$/ isolcpus=2,3"/' /etc/default/grub && sudo update-grub
sudo reboot
kubectl label node k8s-host-01 openebs.io/engine=mayastor
kubectl label node k8s-host-02 openebs.io/engine=mayastor
kubectl label node k8s-host-03 openebs.io/engine=mayastor
kubectl label node k8s-host-01 openebs.io/csi-node=mayastor
kubectl label node k8s-host-02 openebs.io/csi-node=mayastor
kubectl label node k8s-host-03 openebs.io/csi-node=mayastor
helm repo add openebs https://openebs.github.io/openebs
helm repo update
helm install openebs --namespace openebs openebs/openebs --create-namespace
kubectl mayastor upgrade -n openebs --set 'io_engine.coreList={2,3}'
./scripts/setup.sh status
PCI_ALLOWED=0000:06:00.0 ./scripts/setup.sh
PCI_ALLOWED=0000:05:00.0 ./scripts/setup.sh

Create Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mqstore-storage-class
  namespace: backtesting
parameters:
  protocol: nvmf
  repl: "3"
  fsType: xfs
  thin: "true"
provisioner: io.openebs.csi-mayastor
allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
  - key: openebs.io/engine
    values:
    - mayastor

Create Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binance-futures-volume-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 800Gi
  storageClassName: mqstore-storage-class

Please see Slack conversation on Kubernetes/OpenEBS here: https://kubernetes.slack.com/archives/CUAKPFU78/p1723927716943629

Expected behavior Volume class to be created

Screenshots If applicable, add screenshots to help explain your problem.

OS info (please complete the following information):

• Distro: Ubuntu 24.04 LTS
• Kernel version: 6.8.0-38-generic
• MayaStor revision or container image: 2.7.0

Additional context Add any other context about the problem here.

[dmonoquant]

the Issue, was due to having thin: "true" on the StorageClass.

[dmonoquant]

the Issue, was due to having thin: "true" on the StorageClass.

[Abhinandan-Purkait]

@dmonoquant Do you still have the cluster around? Can you send the output for these? io-engine-client bdev list by exec into io-engine container on the io-engine pods in the nodes where the pool committed was showing up as 9TB

[tiagolobocastro]

Also io-engine-client replica list