StarRocks / starrocks-connector-for-apache-flink

Apache License 2.0
189 stars 152 forks source link

support MySQL's entire database synchronization without many Flink SQL #332

Open chenhaifengkeda opened 7 months ago

chenhaifengkeda commented 7 months ago

At Present, synchronizing mysql data in real time takes two steps. Firstly, synchronize database & table schema with SMT. Secondly, synchronize data with Flink SQL. There are three areas for improvement in this method of data synchronization. First of all, flink SQL does not support dynamic schema change during data synchronization. After that, too many binlog parsing clients put a lot of pressure on MySQL servers. Finally, it is very troublesome for users that using two steps to complete real-time data synchronization. A new data synchronization method is needed to optimize the above three problems.