pageserver: circuit breakers

[jcsp]

Examples of recent issues in other services that show up as storage issues:

• Logical replication writes causing >1000x storage amplification, and cause the pageserver to exhaust disk
• Too-fast retries on non-attached tenants saturate our ability to write log errors, and cause the pageserver to become unresponsive.
• Corrupt pages cause compactions on damaged tenants to retry forever, emitting an endless stream of logs

While we of course fix these issues at the source, we can still make the pageserver more resilient.

Implementation:

• Circuit breaker type has a "success()" and "fail()" function, a threshold for how many failures cause the circuit to break, and a duration after which the circuit should close again after breaking (may be none to stay open forever).
• Tenant config property to override the circuit breaker, so that if a seriously damaged tenant's repair requires the pageserver to allow it to operate for some period.
• Add circuit breakers in key locations:
  • Track success/fail of getpage requests for a timeline, and on more than N consecutive errors, break the circuit such that we do no work (including not logging anything, and not taking TenantManager locks) for subsequent requests on that timeline's page_service.
  • Track success/fail after compaction, and on more than N consecutive errors (e.g. 5), and breaking the circuit stops subsequent attempts at compaction for a long period (e.g. 1hr)
  • Periodic check for pathological storage amplification, where one failure is enough to break the circuit, and breaking the circuit stops wal ingest

[jcsp]

Starting point PR that needs generalizing somewhat: https://github.com/neondatabase/neon/pull/6734

The main gate for landing something like this is to have the tenant config mechanism to skip the checks, so that if we need to permit a misbehaving tenant to keep ingesting in order to get it to a fixed state, we can.