[DocDB] Server shutdown path: reject new requests, flush all tablets

[mbautin]

Jira Link: DB-6179

Description

This is an idea mentioned by @rao-vasireddy. To reduce the number of WALs to replay on restart, do the following:

• Blacklist the server being upgraded/restarted (move leaders elsewhere) -- we might already be doing this.
• Set a flag that would result in rejecting all new requests to this server that might add more load to it.
• Flush all tablets (RocksDB, tablet superblocks).
• Shut down using kill -9 (no clean shutdown requirements, unlike what we do in unit tests).

This will reduce the tablet bootstrap cost on restart.

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.

[Huqicheng]

if we make clean shutdown quiesce raft, so no new ops come in wait for current ops to wrap up, so stuff in flight doesn’t get to add new entries to WAL rollover the WAL bonus: maybe don’t even start a new WAL?

[Huqicheng]

Created separate diffs to track clean shutdown related items:

• https://github.com/yugabyte/yugabyte-db/issues/18315
• https://github.com/yugabyte/yugabyte-db/issues/18314
• https://github.com/yugabyte/yugabyte-db/issues/18312
• https://github.com/yugabyte/yugabyte-db/issues/20159

Low priority:

• https://github.com/yugabyte/yugabyte-db/issues/20708