jamadden
commented
3 years ago It might also be possible to generally scope the critical phase a bit tighter, perhaps even taking it a bit sooner, which may help as well.
I think expanding the critical phase to cover the entirety of the TPC protocol is a non-starter in concurrent environments, especially with mixed read/write loads. If it succeeds in locking other greenlets out, then write performance will plummet --- we'd only be capable of committing a single transaction per process at a time, instead of being able to do it mostly in parallel. Worse, uncached read-performance would also plummet, because writing would block those reads.
For those databases+drivers that support atomic (non-interleaved, i.e., requiring no Python intervention to release database locks) lock+move (e.g., PostgreSQL + psycopg2).
If there were conflicts to resolve, loading old states from the database, resolving conflicts, and storing the resolved state back to the database happens in the critical section. The idea is to minimize how long rows are locked by trying to prevent switching to other greenlets.
This critical section carries over to the acquiring the final global commit lock, moving the rows into place, and committing the transaction.
But if that's atomic, it shouldn't need to be in the critical section, since letting other greenlets run once the (small) request has been sent to the database won't affect how long rows are locked. We can let other greenlets make forward progress during that interval by releasing the commit lock.