apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.73k stars 4.58k forks source link

[Bug] [Task] Clickhouse remote function not work as well #16146

Closed smileLuckBoy closed 3 months ago

smileLuckBoy commented 3 months ago

Search before asking

What happened

image I use clickhouse function remote to sync remote table data into local clickhouse , but the count of remote table is not equals as local table , the sql just like insert into dim_call_task select * from remote('host:9000','db','table','userName','password')

What you expected to happen

the count of remote table is equals as local table

How to reproduce

version 3.2.1 ,Deploy using docker there should be two clickhouse ,they have same table A then use the taskType SQL , and choose databaseType clickhouse and then fill the clickhouse insert remote script

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

Code of Conduct

smileLuckBoy commented 3 months ago

the version of clickhouse-jdbc in dolphinscheduler is 0.4.6 (I found it in the dir /opt/dolphinscheduler/libs/worker-server),
I use this version jar to test the clickhouse remote function, and it work well but the dolphinscheduler work not well

smileLuckBoy commented 3 months ago

the max row of sync data is 16384

smileLuckBoy commented 3 months ago

fix, add SETTINGS max_result_rows=0 config after insert remote sql