Don't require batch priority throttling during process failures

[ajbeamon]

The batch priority queue thresholds are currently set such that they are lower than the transaction log spill threshold, which means that batch priority transactions will be stopped during process failures. This was chosen such that users would have the freedom to run heavy batch workloads while minimizing the risk of that workload pushing us into constant spilling, which historically had been a non-trivial performance burden.

Because process failures aren't particularly rare, it does mean that batch priority work may be being throttled too frequently. It's not necessarily bad to have this type of work be stopped occasionally, such that it demonstrates the contract that we provide (i.e. we give the client experience with operating in this mode to confirm they handle it properly), but done too frequently it can hinder the ability to get batch work done.

With recent changes to the spilling logic, it may be the case that we would no longer be concerned with constant spilling due to a heavy batch load and we could adjust the thresholds accordingly. Alternatively, we could change the spill policy to better reflect the fact that we don't need to spill in response to a saturating load (relying instead on ratekeeper to push back).

[xumengpanda]

If we let batch priority workload to keep spilling, users may keep pushing batch workload into the cluster and pushing the spilling logic to reach the maximum capacity and trigger throttling for all workloads. This means: batch priority can interfere the normal transactions.

I think the alternative way is a better approach to prevent the batch priority txn's interference to normal txns. (Of course, it is more complex.)

[xumengpanda]

In case we allow batch priority to spilling, we need to be careful the batch priority workload won't fill up tLog's disk space.