The DistSender circuit breakers will cancel in-flight requests when the breaker trips, such that they can be retried on a different replica instead of getting stuck. However, context cancellation may be ineffective on requests that are processed locally, since it is not respected by disk IO, syscalls, mutex acquisition, etc.
We should see if there is a way to fix this -- a straightforward but likely too expensive option is to spawn a separate goroutine for the request processing, where the client selects on a result channel along with the context channel. Another option is to use e.g. async IO, or lobby the Go team to add context support for various APIs such as IO and mutexes.
We likely can't do anything about this for 24.1, but I'm marking it as a GA blocker for visibility.
The DistSender circuit breakers will cancel in-flight requests when the breaker trips, such that they can be retried on a different replica instead of getting stuck. However, context cancellation may be ineffective on requests that are processed locally, since it is not respected by disk IO, syscalls, mutex acquisition, etc.
We should see if there is a way to fix this -- a straightforward but likely too expensive option is to spawn a separate goroutine for the request processing, where the client selects on a result channel along with the context channel. Another option is to use e.g. async IO, or lobby the Go team to add context support for various APIs such as IO and mutexes.
We likely can't do anything about this for 24.1, but I'm marking it as a GA blocker for visibility.
Jira issue: CRDB-37140
Epic CRDB-39897