zmc
commented
3 years ago When I filed this PR, I also deployed this branch in sepia, watching logs and Sentry carefully. The error rate dropped nearly to zero. The slow trickle of errors after that helped me find a few more places to add retries that I hadn't anticipated (that's what all those force-pushes were).
amathuria
A lot of the times that an attempted transaction is rejected because it can't be properly serialized, an immediate retry of that transaction will succeed. Other times, it may take several attempts. In either case, the transaction time is only extended by a millisecond or so.
Rather than making clients deal with requests failing and having to retry (and potentially losing data), let's try harder to have successful db transactions during the requsest.