apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.81k stars 1.76k forks source link

[Bug] [sqlserver cdc] Caused by: java.lang.UnsupportedOperationException: Don't support SqlSever type 'timestamp' yet, jdbcType:'-2'. #6273

Closed burke-qu closed 6 months ago

burke-qu commented 7 months ago

Search before asking

What happened

sqlserver同步数据到doris报错不支持时间类型, 无法同步含有时间字段的数据表, 但是source 的表时间字段是datetime

SeaTunnel Version

2.3.3

SeaTunnel Config

env {
  # You can set engine configuration here
  execution.parallelism = 1
  job.mode = "STREAMING"
  execution.checkpoint.interval = 10000
}

source {
  SqlServer-CDC {
    result_table_name = "source_p_Project"

    username = "**"
    password = "***"
    database-names = ["dotnet"]
    table-names = ["dotnet.dbo.p_Project"]
    base-url="jdbc:sqlserver://10.129.*.***:1433;databaseName=dotnet"

  }
}

transform {
  sql {
    source_table_name = "source_p_Project"
    query = """ select *, now() as load_time from source_p_Project """
    result_table_name = "trans_p_Project"
  }
}

sink {
        Doris{
                source_table_name = "trans_p_Project"
                fenodes = "10.129.37.206:8030"
                username = "**"
                password = "***"
                table.identifier = "tmp.mys_p_Project_tmp"
                sink.enable-2pc = true
                sink.enable-delete = true
                sink.label-prefix = "p_Projectg_202401231146"
                doris.config = {
                format = "json"
                read_json_by_line = "true"
        }
    }
}

Running Command

./bin/seatunnel.sh --config ./config/sqlserver2doris_1.conf

Error Exception

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.UnsupportedOperationException: Don't support SqlSever type 'timestamp' yet, jdbcType:'-2'.
        at org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.SqlServerTypeUtils.convertFromColumn(SqlServerTypeUtils.java:74)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
        at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
        at org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.SqlServerTypeUtils.convertFromTable(SqlServerTypeUtils.java:88)
        at org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.source.SqlServerIncrementalSource.createDebeziumDeserializationSchema(SqlServerIncrementalSource.java:125)
        at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.prepare(IncrementalSource.java:119)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
        at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
        ... 2 more

Zeta or Flink or Spark Version

zeta

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

CheneyYin commented 7 months ago

It has been fixed in #5668.

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 6 months ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.