apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8k stars 1.81k forks source link

[Bug] [doris] 1.当sink是doris时,varchar类型的数据如果是空会自动加上\N2.自动建表模式提示找不到主键,但是上游source是有主键的 #7314

Closed 13813586515 closed 2 months ago

13813586515 commented 3 months ago

Search before asking

What happened

1.当sink是doris时,varchar类型的数据如果是空会自动加上\N2.自动建表模式提示找不到主键,但是上游source是有主键的,提示必须自己写模版来覆盖

SeaTunnel Version

2.3.6

SeaTunnel Config

env {
  execution.parallelism = 2
  job.mode = "BATCH"
  checkpoint.interval = 10000
}

source {
  Jdbc {
        url = "jdbc:mysql://192.168.0.45:3306/ruoyi-vue-pro?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "lampo_test_2022"
        query = "select * from member_level"
    }
}

sink {
  Doris {
    fenodes = "192.168.0.164:58030"
    username = root
    password = "lampo123"
    database = "ods_crm"
    table = "ods_member_level"
    sink.label-prefix = "ods_crm"
    sink.enable-2pc = "true"
    # sink.enable-delete = "true"
    doris.config {
      format = "json"
      read_json_by_line = "true"
    }

  }
}

Running Command

海豚调度集群模式

Error Exception

ErrorCode:[COMMON-24], ErrorDescription:[The table of ods_crm.ods_member_level has no primary keys, but the template 
     CREATE TABLE IF NOT EXISTS `${database}`.`${table}` (
    ${rowtype_primary_key},
    ${rowtype_fields}
    ) ENGINE=OLAP
     UNIQUE KEY (${rowtype_primary_key})
    DISTRIBUTED BY HASH (${rowtype_primary_key})
     PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "in_memory" = "false",
    "storage_format" = "V2",
    "disable_auto_compaction" = "false"
    ) 
     which has the place holder named ${rowtype_primary_key}. Please use the option named save_mode_create_template to specify sql template]
        at org.apache.seatunnel.common.exception.CommonError.sqlTemplateHandledError(CommonError.java:230)
        at

Zeta or Flink or Spark Version

zeta

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

Hisoka-X commented 3 months ago

Please add table_path in source config. https://seatunnel.apache.org/docs/2.3.6/connector-v2/source/Jdbc#tips

13813586515 commented 3 months ago

Please add table_path in source config. https://seatunnel.apache.org/docs/2.3.6/connector-v2/source/Jdbc#tips

这个是解决了主键的问题了,但是\N的问题没有解决

Hisoka-X commented 2 months ago

@liugddx