[Bug] cdc 3.0.1和flink 1.16.2 集成存在依赖冲入问题

apache / flink-cdc

Flink CDC is a streaming data integration tool

https://nightlies.apache.org/flink/flink-cdc-docs-stable

Apache License 2.0

5.71k stars 1.94k forks source link

[Bug] cdc 3.0.1和flink 1.16.2 集成存在依赖冲入问题 #3071

Closed WuChongYong closed 8 months ago

WuChongYong commented 8 months ago

Search before asking

[X] I searched in the issues and found nothing similar.

Flink version

1.16.2

Flink CDC version

3.0.1

Database and its version

mysql 5.7.26

Minimal reproduce step

guava 依赖冲突，cdc的guava依赖是

org.apache.flink flink-shaded-guava 30.1.1-jre-16.2

flink 1.16.2的guava依赖是

org.apache.flink flink-shaded-guava 31.1-jre-17.0

运行官网样例的api会报类找不到： Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava31/com/google/common/util/concurrent/ThreadFactoryBuilder

如果剔除cdc的guava包，引入31的guava包，也会出现其他类的找不到 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava30/com/google/common/collect/Lists

What did you expect to see?

希望3.0.1之后cdc版本能准确和flink 1.14以上的版本互相兼容

What did you see instead?

但目前3.0.1之后cdc版本能准确和flink 1.16.2 的版本不兼容

Anything else?

No response

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

loserwang1024 commented 8 months ago

@WuChongYong Unlike flink-table-common、flink-connector-base、flink-core whose maven scope is provided , flink-shaded-guava and flink-shaded-force-shading will be included in the final jar package, causing dependency conflict.

Now we can see why dependency conflict occurs:

Since Flink 1.18, version of guava is upgrade from 30 to 31.

Since CDC 3.0, version of guava is also upgrade from 30 to 31
So CDC 3.0 is compatible with Flink 1.18, CDC 2.x is is compatible with Flink 1.13-1.17. CDC 2.x and Flink 1.18 dependency conflict, CDC 3.0 and Flink 1.17 dependency conflict.

Thus, both dependencies should be shaded relocations in pom.xml. By the way, please use English so that foreign developers can also refer to this issue.

@leonardBang , @ruanhang1993 , @PatrickRen CC, WDYT?

YyItRoad commented 8 months ago

和我的一样我用的是flink 1.17.2 和 flink-connector-mongodb-cdc 3.0.1也出现这个问题，然后我将flink-connector-mongodb-cdc降到2.4.2才正常

loserwang1024 commented 8 months ago

@WuChongYong , @YyItRoad , hi, Please give me your code so that I can reproduce it.

WuChongYong commented 8 months ago

you can use the flink 1.18.0 version

---- 回复的原邮件 ---- | 发件人 | Hongshun @.> | | 日期 | 2024年03月06日 20:25 | | 收件人 | @.> | | 抄送至 | @.>@.> | | 主题 | Re: [apache/flink-cdc] [Bug] cdc 3.0.1和flink 1.16.2 集成存在依赖冲入问题 (Issue #3071) |

@WuChongYong , @YyItRoad , hi, Please give me your code so that I can reproduce it.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

PatrickRen commented 8 months ago

Considering collaboration with developers around the world, please re-create your issue in English on Apache Jira under project Flink with component tag Flink CDC. Thank you!

phamvinh1712 commented 5 months ago

Since CDC 3.0, version of guava is also upgrade from 30 to 31

So CDC 3.0 is compatible with Flink 1.18, CDC 2.x is is compatible with Flink 1.13-1.17. CDC 2.x and Flink 1.18 dependency conflict, CDC 3.0 and Flink 1.17 dependency conflict.

Thus, both dependencies should be shaded relocations in pom.xml.

hi @loserwang1024 , would you be able to share an example of pom.xml of how to make this work? I'm trying to use flink 1.16 with flink-cdc 3.0 but getting the same issue above

loserwang1024 commented 5 months ago

Hi, @phamvinh1712 ,

It only influence in test because of maven only load one jar. If you put the jar to flink cluster, no error will happen because the class path is not same.
I also make a pr in https://issues.apache.org/jira/browse/FLINK-35071