apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.04k stars 1.82k forks source link

psgsql表的id是uuid,执行任务报错 #8066

Open scyCoder opened 1 day ago

scyCoder commented 1 day ago

Search before asking

What happened

使用的是pgsql,需要同步的表的主键id,是uuid,在创建cdc同步任务报错: SeaTunnelRuntimeException: ErrorCode:[COMMON-07], ErrorDescription:['SeaTunnel' unsupported data type 'uuid' of 'id'] 想咨询一下,是不支持uuid的类型吗?还是有其它方式,可以进行转换

SeaTunnel Version

版本:2.3.5

SeaTunnel Config

env {
  execution.parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 1000
  read_limit.bytes_per_second = 7000000
  read_limit.rows_per_second = 400
}

source {
  Postgres-CDC {
    username = "xxx"
    password = "xxx"
    database-names = ["xxx"]
    schema-names = ["yyy"]
    table-names = ["xxx.ddd"]
    base-url = "xxx"
    slot.name = "xxx"
    schema = {
      fields {
        id = "uuid"
      }
    }
  }
}

transform {
  # 如果需要进行数据转换,可以在这里配置
}

sink {
  Jdbc {
    url = "xxx"
    driver = "xxx"
    user = "xxx"
    password = "xxx"
    generate_sink_sql = true
    database = "xxx"
    schema = "xxx"
    table = "zzz"
    fields = ["id"]
    batch_size = 1000  # 可选参数:设置批量大小
    isolation_level = "READ_COMMITTED"  # 可选参数:设置事务隔离级别
    primary_keys = ["id"]
  }
}

Running Command

./bin/seatunnel.sh --config ./config/xxx

Error Exception

at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Postgres-CDC'.
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:332)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
        ... 2 more
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-07], ErrorDescription:['SeaTunnel' unsupported data type 'uuid' of 'id']
        at org.apache.seatunnel.common.exception.CommonError.unsupportedDataType(CommonError.java:95)
        at org.apache.seatunnel.api.table.catalog.SeaTunnelDataTypeConvertorUtil.parseComplexDataType(SeaTunnelDataTypeConvertorUtil.java:128)
        at org.apache.seatunnel.api.table.catalog.SeaTunnelDataTypeConvertorUtil.deserializeSeaTunnelDataType(SeaTunnelDataTypeConvertorUtil.java:52)
        at org.apache.seatunnel.api.table.catalog.schema.ReadonlyConfigParser$FieldParser.parse(ReadonlyConfigParser.java:97)
        at org.apache.seatunnel.api.table.catalog.schema.ReadonlyConfigParser$FieldParser.parse(ReadonlyConfigParser.java:84)
        at org.apache.seatunnel.api.table.catalog.schema.ReadonlyConfigParser.parse(ReadonlyConfigParser.java:64)
        at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.buildWithConfig(CatalogTableUtil.java:198)
        at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:108)
        at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:96)
        at org.apache.seatunnel.connectors.seatunnel.cdc.postgres.source.PostgresIncrementalSourceFactory.lambda$createSource$1(PostgresIncrementalSourceFactory.java:91)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
        ... 7 more

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

CosmosNi commented 1 day ago

id = "string"