search

openebs / dynamic-localpv-provisioner

Dynamically deploy Stateful Persistent Node-Local Volumes & Filesystems for Kubernetes that is provisioned from simple Local-Hostpath /root storage.
https://openebs.io
Apache License 2.0
143 stars 63 forks source link

Add ability to check whether PVC storage requirements are fulfilled by the node #123

Open maunavarro opened 3 years ago

maunavarro commented 3 years ago

Describe the problem/challenge you have

It seems that a PVC can claim and be provisioned more storage than the disk size of a particular node. For example, the following PVC requesting 5G should be denied if the available capacity of nodes in cluster is 1G

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-hostpath-pvc
  namespace: openebs
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5G

As it currently stands, the above PVC will be provisioned and bound to a PV .

Describe the solution you'd like

We would like to limit the storage policy to existing free storage in node.

Environment:

trathborne commented 2 years ago

As mentioned in a Slack #openebs thread:

There are a few dimensions to the problem:

Affinity-by-IOPS might be another way to schedule, and as I understand it affinities can have some ranking metric so I can say 'Allocate this PV where:' ... 'it just fits' or 'there is the most space' or 'where other local PVs are generating fewer IOPS'.

niladrih commented 2 years ago

The hostpath provisioner volumes are thin-provisioned (basically how the filesystem handles it). Taking an example... Let's say there's a filesystem with 10 GiB storage capacity. A hostpath volume with 7 GiB capacity request is created on this filesystem. It succeeds. The application process consumes a small amount of space, say 1 GiB. Going forward, if another volume request for say, 5 GiB is created, it will succeed. As the available capacity for the filesystem will be observed to be (10-1) GiB = 9 GiB! The hostpath provisioner does not pre-allocate the requested capacity.

@maunavarro @trathborne -- Does the above hostpath provisioner behaviour suffice your use-case? Thick provisioning is beyond the scope of the hostpath provisioner's design limitations.

trathborne commented 2 years ago

This is working fine for me because I am using XFS quotas and I am ok with overcommitting.

I am more concerned about balance. Consider this case: I have 3 nodes, each with 10GiB of storage. 4 PVCs for 6GiB of storage come in. 2 nodes get 1 PVC, one node gets 2 PVCs. Now another PVC for 6GiB comes in. What is to guaranteed that it does not end up on the node that is already overcommitted?

niladrih commented 2 years ago

@trathborne -- The LVM-based LocalPV CSI-driver is better suited for capacity-aware scheduling use-cases. Here's a link: github.com/openebs/lvm-localpv

trathborne commented 2 years ago

@niladrih I was going to say that LVM doesn't let me overcommit, but then I found https://github.com/openebs/lvm-localpv/blob/2d5196996e97edb82d1840900416835b0b729328/design/monitoring/capacity_monitor.md#thin-logical-volume ... thanks!

niladrih commented 5 months ago

Hi @maunavarro, this issue has been open for a while. Are you are still facing this issue?

This feature is one which is typically present in a CSI driver based storage plugin (e.g. LocalPV ZFS, or LocalPV LVM). This is unfortunately beyond the scope of the out-of-tree provisioner library that this hostpath provisioner uses.

That being said, my viewpoint might be pigeonholed. Please feel free to open a design proposal or continue the discussion here. Marking this as 'wont-fix' for now.

avishnu commented 1 month ago

@niladrih if there is no possible way to achieve this in the external provisioner, should this be closed?

niladrih commented 1 month ago

A LocalPV Rawfile style pre-allocated volume comes to mind.

Another alternative could be filesystem quota.