cloud_storage: cancel/limit reader fibers when clients timeout RPCs

[jcsp]

If a system is overloaded, such that tiered storage reads (including timequeries) are not completing promptly, then Kafka clients will tend to close their connections and issue another request.

When a tiered storage read is in flight, we do not cancel it when the original Kafka request's client closes their connection: it will remain enqueued and run to completion.

On an overloaded system, this can lead to unbounded growth in the number of reads in flight.

There are a couple of angles to addressing this:

• We should avoid initiating reads if the queue is already so long
• We should apply an internal timeout to enqueued cloud storage operations
• The rpc/Kafka layer should pass an abort source down into the cloud storage layer, so that when a client gives up, we can cancel the I/Os.

JIRA Link: CORE-1358

[jcsp]

The more specific case of this for timequeries is https://github.com/redpanda-data/redpanda/issues/10854, but we probably need a solution for all fetch requests.

[BenPope]

Unassigning myself since #12021 doesn't quite get us there.

[VladLazar]

@BenPope could you drop a note on what the gap between the improvements from #12021 and this ticket?

[BenPope]

@BenPope could you drop a note on what the gap between the improvements from #12021 and this ticket?

12021 is just the last bullet point above.

[github-actions[bot]]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[piyushredpanda]

This is still something we need to pursue.