apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.82k stars 1.76k forks source link

[Bug] [seatunnel-connectors-v2] [connector-clickhouse] import data only to one machine where the host has multiple hosts[ host1:8123,host2:8123]] #5552

Closed wuxizhi777 closed 11 months ago

wuxizhi777 commented 11 months ago

Search before asking

What happened

import data only to one machine where the host has multiple hosts[ host1:8123,host2:8123]]

where the table in clickhouse-sink config which engine is not set to the type of 'Distributed' 。 the data import only to the one machine , in this case the machine is host1 。 the machine nameed host2 have not any data

SeaTunnel Version

seatunnel 2.3.1

SeaTunnel Config

{
    "filter" : {},
    "transform" : [],
    "sink" : [
       {
             "database" : "damaiX",
             "password" : "",
             "save_mode" : "overwrite",
             "clickhouse.socket_timeout" : 100000,
             "host" : "host1:8123,host2:8123",
             "bulk_size" : 5000000,
       "fields" : [
                 "id",
                 "gazj",
                 "dt"
              ],
             "plugin_name" : "clickhouse",
             "table" : "adm_dmp_user_id_replica",
             "username" : ""
     }
    ],
     "source" : [
        {
            "result_table_name" : "table_name",
            "pre_sql" : "select id,gazj, dt from table ",
             "plugin_name" : "hive"
         }
        ],
    "env" : {
       "spark.executor.memory" : "28g",
       "spark.sql.catalogImplementation" : "hive",
       "spark.default.parallelism" : 72,
       "spark.streaming.batchDuration" : 5,
       "spark.app.name" : "XXXXX.app.name",
       "spark.ui.port" : 13000,
       "spark.executor.cores" : 4,
       "spark.executor.instances" : 9,
       "spark.sql.shuffle.partitions" : 72,
       "spark.dynamicAllocation.enabled" : false
   }
 }

Running Command

sh ~/bin/setart-seatunnel-spark.sh --master yarn --deploy-mode client -config  xxx.config

Error Exception

function error

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

Carl-Zhou-CN commented 11 months ago

@wuxizhi777 This is probably as expected,Similar problem link #5435

wuxizhi777 commented 11 months ago

ok. I will close the issue