sync data speed is too low, please help me

[zhangtao106]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I had searched in the issues and found no similar issues.

Describe the proposal

I was synchronized data from DRDS to Doris, my query range is “id < 50000000”, but total data less than 50000000, when i synchorized, that speed was very slow! less than datax much .
 my data  in DRDS has 42  cloumns, and this is my conf:

env { checkpoint.interval = 10000 job.mode = "batch" } source{ Jdbc { url = "jdbc:mysql://x.x.x.x:3306/pp_data_ds?useSSL=false&serverTimezone=GMT%2B8&useUnicode=true&characterEncoding=utf8&socketTimeout=6000000&waitTimeout=60000&autoReconnect=true&rewriteBatchedStatements=true&connectTimeout=60000" driver = "com.mysql.cj.jdbc.Driver" connection_check_timeout_sec = 100 user = "user_ro" password = "******" query = "selectprovince_name`, ... from table_data where id >0 and id < 50000000"

}

}

sink { Doris { fenodes = "x.x.x.x:8030" username = test password = "***" database = "test" table = "td_sub_sms_port_bk" sink.label-prefix = "test-batch" sink.enable-2pc = "true" sink.enable-delete = "true" doris.config { format = "json" read_json_by_line = "true" } } }`

when I submit job config into cluster, log info is update slow ,

Task list

env { checkpoint.interval = 10000 job.mode = "batch" } source{ Jdbc { url = "jdbc:mysql://x.x.x.x:3306/pp_data_ds?useSSL=false&serverTimezone=GMT%2B8&useUnicode=true&characterEncoding=utf8&socketTimeout=6000000&waitTimeout=60000&autoReconnect=true&rewriteBatchedStatements=true&connectTimeout=60000" driver = "com.mysql.cj.jdbc.Driver" connection_check_timeout_sec = 100 user = "user_ro" password = "**" query = "select province_name, ... from table_data where id >0 and id < 50000000"

}

}

sink { Doris { fenodes = "x.x.x.x:8030" username = test password = "***" database = "test" table = "td_sub_sms_port_bk" sink.label-prefix = "test-batch" sink.enable-2pc = "true" sink.enable-delete = "true" doris.config { format = "json" read_json_by_line = "true" } } }

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[panpan2019]

I can fix

[Anoye]

I encountered the same problem, have you solved it yet

[richardblabin]

I can fix

how? more specific please.thanks a lot