search

tikv / sig-transaction

Resources for the transaction SIG
62 stars 13 forks source link

Tracking issue: async commit #36

Closed nrc closed 3 years ago

nrc commented 4 years ago

Design docs, etc are in this repo in design/parallel-commit

All open issues

Open PRs:

Work in progress:

Plan and status

Demo-able version

Goal is to demonstrate async commit which works and has roughly the expected performance profile, though not be performant. Some corner cases may not be correct. Demonstrate that the work is feasible and beneficial.

Currently in implementation.

Outstanding issues:

5.0 version

Address all outstanding issues. Handle corner cases and tools. Test, benchmark, and optimise.

Currently in implementation.

Outstanding issues:

Further work

Not blocking 5.0 release.

nrc commented 4 years ago

Increment 1

Async commit protocol is implemented and functional. Implementation (especially in TiKV) may not be high performance. There may be correctness issues with edge cases.

Deliverable: can demo async commit with real workloads. Can reliably run sysbench benchmark. Optimistic transactions only.

Deadline: 2020-08-07

Increment 2

Use memory locks in TiKV and avoid PD calls

Support pessimistic transactions

Increment 3

All correctness edge cases addressed

nrc commented 4 years ago

Current blocked on increment 1 is implementing checking of secondary locks in TiDB (https://github.com/pingcap/tidb/pull/18467, but needs significant changes). That is blocked on updating kvproto in TiDB due to back-compat issues, fixed by https://github.com/pingcap/kvproto/pull/659

tisonkun commented 3 years ago

Design docs, etc are in this repo in design/parallel-commit

All open issues

  • Audit uses of commit ts (to support future non-unique timestamps) #19 - seems that only issues are with tools now
  • Replica read #20
  • Syncing timestamps #21
  • Testing plan #23
  • Problems about usage of max_read_ts #45
  • Schema check issue #51
  • Async commit does not ensure linearizability tikv/8589

Open PRs:

Work in progress:

  • QA and testing (#23, @youjiali1995, @zyguan)
  • Handle non-unique timestamps (@MyonKeminta)
  • TLA+ model (@andylokandy)
  • Manual testing (@sticnarf )
  • Benchmarking

Plan and status

Demo-able version

Goal is to demonstrate async commit which works and has roughly the expected performance profile, though not be performant. Some corner cases may not be correct. Demonstrate that the work is feasible and beneficial.

Currently in implementation.

Outstanding issues:

5.0 version

Address all outstanding issues. Handle corner cases and tools. Test, benchmark, and optimise.

Currently in implementation.

Outstanding issues:

  • binlog (pingcap/tidb#18622 (comment))
  • optimisation of concurrency manager (more benchmarking required to be sure, @sticnarf)
  • choose default value for max keys

Further work

Not blocking 5.0 release.

hi @nrc could you please update the link to our design doc from design/parallel-commit to design/async-commit. Note that the original link is broken because feature name changed and also the file layout.