storage: reject writes in advance of out-of-disk

[jbowens]

A CockroachDB store may run out of disk space because the operator has insufficient monitoring, an operator is unable to respond in time, etc. A CockroachDB store that exhausts available disk space crashes the node. This is especially problematic within CockroachCloud, where Cockroach Labs SREs are responsible for the availability of CockroachDB but have no control over the customer workload.

Recovering from disk space exhaustion is tricky. Deleting data within CockroachDB requires writing tombstones to disk and writing new immutable sstables before removing old ones. Adding new nodes also requires writing to existing nodes. The current recommended solution is to reserve a limited amount of disk space in a ballast file when initializing stores. If a store exhausts available disk space, the ballast file may be manually removed to provide some headroom to process deletions.

CockroachDB 21.2 introduced automatic ballasts that help ensure that operators have headroom for manual recovery from an out-of-disk node/cluster. (See #66493) It does not altogether prevent node crashes.

---

This issue tracks work related to the goal of automatically handling a replication group that's under-provisioned, and avoiding resulting node crashes. If a CockroachDB is writing more data to a particular zone configuration than there is storage available to that zone configuration, eventually writes will fail: Today, they will fail because of a quorum of nodes will crash. We'd like to support a more graceful degradation that avoids crashing the nodes, rejects incoming writes to stave off total disk space exhaustion and allows certain queries that allow reclamation of disk space.

Jira issue: CRDB-11920

[erikgrinaker]

I think it could make sense to do this in admission control. Wrote up a separate proposal in #79210 instead of hjacking this issue, since we'd still need to address out-of-disk if we don't want to do it in admission control.

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!