backupccl: ensure each ExportRequest returns on a range boundary

[adityamaru]

When a backup sends ExportRequests it does so with a TargetBytes set to 1 to ensure that only a single SST file is returned in the response. This was also what we subconsciously relied on to limit the request only going to a single range. In #2554 we saw a unique situation where the span covered by the ExportRequest spanned several ranges and the first several ranges were empty and so a single client call led to ~50 DistSender Send calls. Each ExportRequest is given 5mins by default to complete, and this timeout was added under the impression that the request would only hit one range. Granted processing an empty range should be very quick but when you have ~6 concurrent workers sending these requests to a store during a backup you can quickly clog up the concurrent request limiter. This is likely what caused backups to frequently fail in the issue linked above.

When we plan a backup, before executing the flow, we get a span -> node mapping where each span belongs to a single range and does not cross range boundaries. In the backup processor however we lose this per range chunking as we collapse these spans such that they could span range boundaries when sent as an ExportRequest. We should maintain the contract that a single client side ExportRequest should only touch one range, even if that range ends up being empty. I see two options here:

• We can set ReturnOnRangeBoundary on the ExportRequest. I don't think this would work out of the box for empty ranges because we still check that our replyKeys > 0 - https://github.com/cockroachdb/cockroach/blob/c396820220a0287c842978f31d1dc863c0bb0314/pkg/kv/kvclient/kvcoord/dist_sender.go#L1711 but I wonder if we could get rid of that check there? I don't see any other users of this header field on master.

• We can teach the backup processor to read from a range descriptor and truncate the EndKey to be within a range boundary. The remaining span can then be put on the worker queue to be picked up in the next iteration.

Jira issue: CRDB-33408

[blathers-crl[bot]]

cc @cockroachdb/disaster-recovery

[blathers-crl[bot]]

Hi @dt, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[dt]

Completed and backported (default off) to 23.1.