Support direct write into TiFlash without going through TiKV

[ilovesoup]

Description Support direct write into TiFlash without TiKV replica.

Category Feature

Value Enable a lot more pure columnar format oriented scenarios without bothering storing as row format first.

Task List

• [x] Study in what form the final product will be delivered:
  • [x] Upstream being TiDB or other db?
  • [x] Any ecosystem tools to involve? I.e. DM? Lightning?
  • [x] Is a minimal TiKV required? At what resource requirement? I.e. hardware? latency/throughput?
• [ ] Study the feasibility of TiFlash being Raft leader to accept direct write:
  • [x] Any PD replacement rule changes required?
  • [ ] Use TiFlash proxy VS. TiKV Raft layer + engine trait?
  • [x] TiFlash needing to be follower or just learner?
• [ ] Performance in terms of transaction and consistency consideration due to properties of TiFlash columnar store:
  • [ ] Percolator VS. single row transaction (1PC)?
  • [ ] Strong consistency VS. eventual consistency?
  • [ ] Update/delete needed?
  • [ ] Latency & throughput requirements?
  • [ ] Batch write?
  • [ ] TiDB SQL statement customization?
• [ ] Minimal viable product (MVP).

Progress Tracking Project Link

Time

GanttStart: 2020-07-01 GanttDue: 2020-12-28 GanttProgress: 50%

[HunDunDM]

/label component/deployment

[zanmato1984]

Related project: https://github.com/pingcap/tics/projects/15

[zanmato1984]

Description Support direct write into TiFlash without TiKV replica.

Category Feature

Value Enable a lot more pure columnar format oriented scenarios without bothering storing as row format first.

Task List

• [ ] Study in what form the final product will be delivered:
  • [ ] Upstream being TiDB or other db?
  • [ ] Any ecosystem tools to involve? I.e. DM? Lightning?
  • [ ] Is a minimal TiKV required? At what resource requirement? I.e. hardware? latency/throughput?
• [ ] Study the feasibility of TiFlash being Raft leader to accept direct write:
  • [ ] Any PD replacement rule changes required?
  • [ ] Use TiFlash proxy VS. TiKV Raft layer + engine trait?
  • [ ] TiFlash needing to be follower or just learner?
• [ ] Performance in terms of transaction and consistency consideration due to properties of TiFlash columnar store:
  • [ ] Percolator VS. single row transaction (1PC)?
  • [ ] Strong consistency VS. eventual consistency?
  • [ ] Update/delete needed?
  • [ ] Latency & throughput requirements?
  • [ ] Batch write?
  • [ ] TiDB SQL statement customization?
• [ ] Minimal viable product (MVP).

Progress Tracking Project Link

Time

GanttStart: 2020-07-01 GanttDue: 2020-12-31

[scsldb]

Reason for reprioritization: there are more important development tasks at hand.