tidb-incubator / TiBigData

TiDB connectors for Flink/Hive/Presto

Apache License 2.0

216 stars 56 forks source link

Roadmap 2022 #134

Open zhangyangyu opened 2 years ago

zhangyangyu commented 2 years ago

This is TiBigData roadmap 2022.

It's under iterative edition and we welcome everyone to make advice.

Features

Flink

[ ] Fast Lookup with PK & Secondary Indices
[ ] Support Pre Split & Auto Random & Auto Increment when batch load
[x] Support Upsert & Delete DML
[ ] Unified Batch & Streaming Processing
[ ] Support Catalog
[ ] Support External Connector Interface
[ ] Support FDW integration and Cloud EMR integration
[ ] Support Async Query
[ ] SSL Support
[x] Telemetry

Spark (we'd like to merge TiSpark into TiBigData)

[ ] Support Pre Split & Auto Random & Auto Increment when batch load
[x] Support Upsert & Delete DML
[ ] Unified Batch & Streaming Processing
[x] Support Catalog
[ ] Support External Connector Interface
[ ] Support FDW integration and Cloud EMR integration
[ ] Support Async Query
[x] SSL Support
[ ] Telemetry

Quality

[x] Establish GA criteria

humengyu2012 commented 2 years ago

What's the Async Query?

sunxiaoguang commented 2 years ago

What's the Async Query?

It's about using FDW to submit asynchronous query to Spark / Flink from TiDB directly.

gbzy commented 2 years ago

It's important to support tikv native writing, the writing speed is the bottleneck of flink batch job.

sunxiaoguang commented 2 years ago

It's important to support tikv native writing, the writing speed is the bottleneck of flink batch job.

This particular feature is what you want

Support Upsert & Delete DML