search

pingcap-inc / tidb2dw

Replicate data changes from TiDB to Data Warehouses in real-time
MIT License
13 stars 7 forks source link

tidb2dw should only has dependency on shared storage #101

Open zhangjinpeng87 opened 7 months ago

zhangjinpeng87 commented 7 months ago

Right now the tidb2dw will connect to upstream TiDB to fetch some information like schema and column default value which break the simple dependency principle. The tidb2dw service should only have dependcy on the shared object storage like S3 and GCS, in this way, there should not have upstream database permission issue and network dependency issues since upstream clusters might in different VPC networks.

tidb -> ticdc ----->
                    --------> S3/GCS ------->  
                                              -------> tidb2dw -------> Snowflake/Redshift/...
                    --------> S3/GCS ------->
tidb -> ticdc ----->