kafka/cloud_storage: fetch requests should acquire enough reader units for all their reads before starting any

[jcsp]

Kafka fetch and timequery operations require doing I/O on a list of partitions. This can be a long list, if it's on a topic with 1000 partitions.

Since we limit the concurrency of tiered storage reads (https://github.com/redpanda-data/redpanda/pull/11914), if there are two separate client requests that need 1000 units each, and we only have 1000 units to share, then each request might find that it can do 500 I/Os, but are blocked reading the other 500 I/Os. If I/O is slow, then both requests might time out on the client side -- a "nobody wins" compromise. Clients will retry but the same thing can happen again.

To fix this, the acquisition of units on the reader semaphore needs to move up a level into the kafka layer, where it should acquire enough units for all the concurent reads it intends to do, and then pass those units down into remote_partition when calling make_reader. That way, in the example above, one request would succeed (get all 1000 units), and the other would fail. The failed on can then retry, and eventually the client will be satisfied, rather than being stuck in a failure loop.

JIRA Link: CORE-1366

[github-actions[bot]]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[github-actions[bot]]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.