search

apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.61k stars 1.9k forks source link

[Bug] FlinkCDC读取SqlServer异常 #2383

Closed sprybee closed 5 months ago

sprybee commented 1 year ago

Search before asking

Flink version

Flink - 1.15.0

Flink CDC version

SqlServer-Connector 2.4.0

Database and its version

SqlServer-Version :[ Microsoft SQL Server 2016 (SP2-CU17) (KB5001092) - 13.0.5888.11 (X64) ]

Minimal reproduce step

create_source_table.id => id : bigint identity & PRIMARY KEY (id) NOT ENFORCED

What did you expect to see?

Don't support SqlSever type 'bigint identity' yet, jdbcType:'-5'.

org.apache.flink.util.FlinkRuntimeException: Generate Splits for table error at com.ververica.cdc.connectors.sqlserver.source.dialect.SqlServerChunkSplitter.generateSplits(SqlServerChunkSplitter.java:125) ~[flink-connector-sqlserver-cdc-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.base.source.assigner.SnapshotSplitAssigner.getNext(SnapshotSplitAssigner.java:178) ~[flink-cdc-base-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.base.source.assigner.HybridSplitAssigner.getNext(HybridSplitAssigner.java:137) ~[flink-cdc-base-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator.assignSplits(IncrementalSourceEnumerator.java:174) ~[flink-cdc-base-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator.handleSplitRequest(IncrementalSourceEnumerator.java:97) ~[flink-cdc-base-2.4.1.jar:2.4.1] at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$handleEventFromOperator$2(SourceCoordinator.java:230) ~[flink-runtime-1.15.0.jar:1.15.0] at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$9(SourceCoordinator.java:406) ~[flink-runtime-1.15.0.jar:1.15.0] at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40) [flink-core-1.15.0.jar:1.15.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_291] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_291] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_291] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_291] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291] Caused by: java.lang.UnsupportedOperationException: Don't support SqlSever type 'bigint identity' yet, jdbcType:'-5'. at com.ververica.cdc.connectors.sqlserver.source.utils.SqlServerTypeUtils.convertFromColumn(SqlServerTypeUtils.java:75) ~[flink-connector-sqlserver-cdc-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.sqlserver.source.utils.SqlServerTypeUtils.fromDbzColumn(SqlServerTypeUtils.java:31) ~[flink-connector-sqlserver-cdc-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.sqlserver.source.dialect.SqlServerChunkSplitter.fromDbzColumn(SqlServerChunkSplitter.java:144) ~[flink-connector-sqlserver-cdc-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.base.source.assigner.splitter.JdbcSourceChunkSplitter.isEvenlySplitColumn(JdbcSourceChunkSplitter.java:118) ~[flink-cdc-base-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.sqlserver.source.dialect.SqlServerChunkSplitter.splitTableIntoChunks(SqlServerChunkSplitter.java:194) ~[flink-connector-sqlserver-cdc-2.4.1.jar:2.4.1] at com.ververica.cdc.connectors.sqlserver.source.dialect.SqlServerChunkSplitter.generateSplits(SqlServerChunkSplitter.java:95) ~[flink-connector-sqlserver-cdc-2.4.1.jar:2.4.1] ... 14 more

What did you see instead?

不支持bigint identity 看了下源码确实没有 ，求破 image 没有-5

Anything else?

No response

Are you willing to submit a PR?

GOODBOY008 commented 1 year ago

@sprybee Please ping me after your pr is ready.Thank you~

PatrickRen commented 5 months ago

Considering collaboration with developers around the world, please re-create your issue in English on Apache Jira under project Flink with component tag Flink CDC. Thank you!