apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.82k stars 1.76k forks source link

sink为clickhouse,插入到clickhouse后,字段对应的数据不一致 #6274

Closed Seapein closed 7 months ago

Seapein commented 7 months ago

Search before asking

What happened

source为mysql,sink为clickhouse时,使用FieldMapper,如果mysql与clickhouse的表字段顺序不同,插入到clickhouse后,字段对应的数据会不一致。 89158b637a3adb5fe4d324a04cc8cf8

SeaTunnel Version

2.3.3

SeaTunnel Config

env {
"job.mode"=BATCH
"job.name"="SeaTunnel_Job"
}
source {
Jdbc {
    "connection_check_timeout_sec"=30
    parallelism=1
    "result_table_name"=Table12372182520352
    query="SELECT `id`, `code`, `name` FROM `admin_langfang`.`depttt`"
    driver="com.mysql.cj.jdbc.Driver"
    password="123456"
    url="jdbc:mysql://192.168.1.191:3306/seatunnel?useUnicode=true&characterEncoding=utf-8&useSSL=false"
    user=root
}
}
transform {
FieldMapper {
    "result_table_name"=Table12372182520353
    "source_table_name"=Table12372182520352
    "field_mapper" {
        id=id
        code=code
        name=name
    }
}
}
sink {
Clickhouse {
    username=default
    password="keli@1991"
    "clickhouse.config" {}
    "bulk_size"=20000
    "split_mode"="false"
    "support_upsert"="false"
    "allow_experimental_lightweight_delete"="false"
    "source_table_name"=Table12372182520353
    database="inside_data"
    table="depe_test"
    host="192.168.1.207:8123"
    url="jdbc:clickhouse://192.168.1.207:8123/inside_data?useSSL=false&useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&autoReconnectForPools=true&connectTimeout=300000&socketTimeout=300000"
    driver="ru.yandex.clickhouse.ClickHouseDriver"
    user=default
}
}

Running Command

./bin/seatunnel.sh --config ./config/test.template

Error Exception

no error

Zeta or Flink or Spark Version

使用zeta

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

Seapein commented 7 months ago

4d6117437dc3d319db8bfb835f0249d mysql数据源截图

liugddx commented 7 months ago

Change field mappings . Refer to https://seatunnel.apache.org/docs/2.3.3/transform-v2/sql

Seapein commented 7 months ago

Change field mappings . Refer to https://seatunnel.apache.org/docs/2.3.3/transform-v2/sql

Hi,I have tried this method before, but it did not take effect: transform { Sql { "result_table_name"=Table12372182520353 "source_table_name"=Table12372182520352 query = "select id as id, code as code, name as name from Table12372182520352 " } }

liugddx commented 7 months ago

Try to adjust the following order.

Seapein commented 7 months ago

Try to adjust the following order.

hi,what order?

liugddx commented 7 months ago

transform { Sql { "result_table_name"=Table12372182520353 "source_table_name"=Table12372182520352 query = "select id as id, name as name,code as code from Table12372182520352 " } }

Seapein commented 7 months ago

transform { Sql { "result_table_name"=Table12372182520353 "source_table_name"=Table12372182520352 query = "select id as id, name as name,code as code from Table12372182520352 " } }

Thank you very much for solving my problem!