Fail writes when storage is about to end

[dorlaor]

Scylla automatically shuts down when there is no space but it will be better to add another line of defense, otherwise it will be a bit complex to bring the node back up while trying to expand the cluster or the storage.

If we'll configure a threshold that beyond that point, all writes queries will fail immediately it can help. We can define it in the yaml as % of overall storage or minimum available storage (since some of the storage may be taken by snapshots, backups, ..).

In addition, since every node is independent, it will be nice to have a state (like a CQL state) which the admin can switch the DB to a read-only mode and thus reach a similar goal

[gleb-cloudius]

Update/Insert are not the only way data can be added to a node. Some reads also require write (lwt). Repair and read repair can also add data to a node.

-- Gleb.

[tzach]

a read-only state per node is problematic, since the node data will become out of date.