apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.81k stars 1.76k forks source link

[Bug]Sqlserver CDC #5449

Closed shuxian6688 closed 12 months ago

shuxian6688 commented 1 year ago

Search before asking

What happened

sqlserver cdc to drois Don't support SqlSever type 'timestamp'

SeaTunnel Version

2.3.3

SeaTunnel Config

env {
  execution.parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 10000
}

source {
  SqlServer-CDC {
    result_table_name = "customers"

    username = "XXXXX"
    password = "XXXXX"
    database-names = ["xxx"]
    table-names = ["xxx.dbo.xxxx"]
    base-url="jdbc:sqlserver://XXXXXX:1433;databaseName=xxxxxx"
  }
}

transform {
  Filter {
    source_table_name = "customers"
    result_table_name = "Acc_Charge"
    fields = [InterID
,JobNoRelOne
,JobNoRelTwo
,AutoHistoryID
  }
}

sink {
    Doris {
        fenodes = "xxxxxx:8030"
        username = root
        password = "xxxxxxxx"
        table.identifier = "example_db.xxxxx"
        sink.enable-2pc = "true"
        sink.label-prefix = "test_csv"
        doris.config = {
          format = "csv"
          column_separator = ","
        }
    }
}

Running Command

"seatunnel.sh","useCustom":true,"deployMode":"local"}

Error Exception

2023-09-08 19:22:50,044 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
    Caused by: java.lang.UnsupportedOperationException: Don't support SqlSever type 'timestamp' yet, jdbcType:'-2'.
        at org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.SqlServerTypeUtils.convertFromColumn(SqlServerTypeUtils.java:74)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
        at org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.utils.SqlServerTypeUtils.convertFromTable(SqlServerTypeUtils.java:88)
        at org.apache.seatunnel.connectors.seatunnel.cdc.sqlserver.source.source.SqlServerIncrementalSource.createDebeziumDeserializationSchema(SqlServerIncrementalSource.java:125)
        at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.prepare(IncrementalSource.java:119)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
        at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
        ... 2 more

Zeta or Flink or Spark Version

1.17.1

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

CheneyYin commented 1 year ago

According to the error log, there was an error parsing the table schema. Would you like to provide the table schema of the source?

CheneyYin commented 1 year ago

datetimeoffset has a similar problem.

shuxian6688 commented 1 year ago

CREATE TABLE [dbo].[Acc_Charge] ( [InterID] varchar(20) COLLATE Chinese_PRC_CI_AS NOT NULL, [JobNoRelOne] varchar(20) COLLATE Chinese_PRC_CI_AS NULL, [JobNoRelTwo] varchar(20) COLLATE Chinese_PRC_CI_AS NULL, [AutoHistoryID] varchar(200) COLLATE Chinese_PRC_CI_AS NULL, [RowVer] timestamp NULL, CONSTRAINT [PK_ACC_CHARGE] PRIMARY KEY NONCLUSTERED ([InterID]) WITH (PAD_INDEX = OFF, FILLFACTOR = 90, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON [PRIMARY] )

EricJoy2048 commented 1 year ago

Yes, this is a bug.

hailin0 commented 12 months ago

Close this issue because https://github.com/apache/seatunnel/issues/5454 already exists