Open andytyi opened 1 week ago
The split column selection order is: 1、primary key and primary key type is tinyint 2、unique key and unique key type is tinyint 3、no tinyint key , compare primary key and unique key type,the type order is: SMALLINT、INT、BIGINT、DECIMAL、STRING This is a design issue,and I think can add user defined split column,if need can assigned to me
YES, I need.
Search before asking
What happened
使用CDC同步MySQL一张表到doris,表数据大概有6千万,在进行数据分割时seatunnel同步任务并没有使用主键(id)进行分割,而是使用唯一索引(ad_type)进行分割,数据同步非常慢。使用show full processlist查看MySQL正在执行的SQL如下:SELECT MIN(
ad_type
), MAX(ad_type
) FROMtest
.abroad_day_recharge
,同步任务一直卡在这个SQL处, 表结构如下: CREATE TABLEabroad_day_recharge
(id
int unsigned NOT NULL AUTO_INCREMENT,union_link_id
int unsigned DEFAULT '0',book_id
bigint unsigned DEFAULT '0',system
tinyint unsigned DEFAULT '1',user_date
int unsigned DEFAULT '0',order_date
int unsigned DEFAULT '0',platform
int unsigned DEFAULT '0',ad_type
tinyint DEFAULT '1',money
int unsigned DEFAULT '0',created_at
timestamp NULL DEFAULT CURRENT_TIMESTAMP,updated_at
timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (id
), UNIQUE KEYuniq_key_word
(union_link_id
,user_date
,order_date
,ad_type
,system
) USING BTREE, KEYidx_user_date
(user_date
), KEYidx_order_date
(order_date
), KEYidx_platform
(platform
) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ciSeaTunnel Version
2.3.5
SeaTunnel Config
Running Command
Error Exception
Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
Code of Conduct