search

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.23k stars 1.3k forks source link

Move write-only transactions to the end of commit batches #3741

Open sfc-gh-tclinkenbeard opened 3 years ago

sfc-gh-tclinkenbeard commented 3 years ago

Write-only transactions can't be aborted due to conflicts (barring self-conflicts), but they can cause unnecessary intra-batch conflicts for other transactions. This problem can be mitigated by moving all transactions with empty read-conflict sets (again, excluding self-conflict keys) to the end of all commit batches. The two main downsides to this approach are:

1) Increases proxy CPU usage. 2) Makes it more likely that transactions from the same client will be committed in a different order than commit is called. This is already a possibility though, so this is not a major weakness.

Given these weaknesses, this may only be a worthwhile change for certain workloads that have lots of write-only transactions and a high conflict-rate.

xumengpanda commented 3 years ago

This is under the umbrella idea of re-ordering transactions in a batch to reduce the conflict rate.

Since this is a small research area that has been studied a lot, such as http://www.vldb.org/pvldb/vol12/p169-ding.pdf at VLDB'19, IMO, if we want to do this type of things, we can sketch out all practically-effective reordering ideas and do them as a project.

I don't object to solving this alone thought. I just prefer to solve it in a systematic way.

sfc-gh-almiller commented 2 years ago

http://pages.cs.wisc.edu/~yxy/pubs/aria.pdf includes a counter argument to a more systematic solution, and shows evidence that the simplistic reordering proposed offers a good degree of throughput improvement under skew.