search

pingcap / tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV
Apache License 2.0
880 stars 244 forks source link

Question about tikv-client, please help me! #2636

Closed MengXiangDing closed 1 year ago

MengXiangDing commented 1 year ago

How does TiSpark obtain TiKV data?

Is it by obtaining the raft log of TiKV?

Is the corresponding cdc event obtained through tikv-client?

@Dieken please help me！thank you！

zhangyangyu commented 1 year ago

TiSpark obtains TiKV data by tikv/client-java in the latest release. For CDC event, currently, there is no official supported client/driver to consume it.

MengXiangDing commented 1 year ago

TiSpark obtains TiKV data by tikv/client-java in the latest release. For CDC event, currently, there is no official supported client/driver to consume it.

@zhangyangyu thank you！I still have a question, how does tikv/client-java obtain the incremental data of TiKV? Is it driven by jdbc?

zhangyangyu commented 1 year ago

no, tikv/client-java cannot obtain the incremental data(the CDC feature). It's not a JDBC driver, it talks to TiKV via GRPC.

MengXiangDing commented 1 year ago

no, tikv/client-java cannot obtain the incremental data(the CDC feature). It's not a JDBC driver, it talks to TiKV via GRPC.

@zhangyangyu ok, thank you for your answer ! What tikv/client-java obtains is the raft log generated by the region leader, can I understand it this way?

shiyuhang0 commented 1 year ago

client-java just read the batch data via RPC provided by TiKV which reads with RocksDB SST file. So, TiSpark does not support streaming now.

MengXiangDing commented 1 year ago

client-java just read the batch data via RPC provided by TiKV which reads with RocksDB SST file. So, TiSpark does not support streaming now.

@shiyuhang0 ok， thank you！I will open an issue here https://github.com/pingcap/tiflow