neondatabase / autoscaling

Postgres vertical autoscaling in k8s
Apache License 2.0
164 stars 21 forks source link

Support disk quotas #1071

Closed petuhovskiy closed 1 month ago

petuhovskiy commented 2 months ago

Problem description / Motivation

We have an issue with some users running out of a limited disk space that we have inside the VMs (https://github.com/neondatabase/cloud/issues/13127).

The chosen solution for this issue is to give everyone a large disk, but limit the disk usage with an ext4 feature called "project quotas".

After we have this feature with the quotas, we can add new logic in the control plane to give users more disk based on the maxCU setting in their endpoint.

Feature idea(s) / DoD

DoD is to get the change from "Implementation ideas" rolled out to staging and production.

Implementation ideas

  1. The EmptyDiskSource struct should get a new field called EnableQuotas, which enables support for project quotas in the filesystem. The corresponding yaml for the disk can look like this:

    disks:
    - emptyDisk:
      discard: true
      size: 100Gi
      enableQuotas: true
    mountPath: /var/db/postgres/compute
    name: pgdata
    readOnly: false
  2. Inside the VM, there should be a new script to control the quotas called /neonvm/bin/set-disk-quota. This script allows to set the quota for the specific filesystem:

    # Usage can be like this:
    /neonvm/bin/set-disk-quota <NEW_SIZE_IN_KB> <MOUNTPOINT>
petuhovskiy commented 1 month ago

Implemented in https://github.com/neondatabase/autoscaling/pull/1062