apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.92k stars 1.79k forks source link

[Bug] [Kafka] Use EXACTLY_ONCE commit error for kafka sink #7755

Open hailin0 opened 2 weeks ago

hailin0 commented 2 weeks ago

Search before asking

What happened

https://mp.weixin.qq.com/s/DSv76j1riGMNBkR0VWNEhg

kafka clientId duplicate for writer and committer

image

SeaTunnel Version

dev

SeaTunnel Config

-

Running Command

-

Error Exception

-

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

corgy-w commented 2 weeks ago

Supplement: Recently,I have also tested zeta, and the corresponding error is as follows. I think this is the zeta error described in the article.(guess) image

liunaijie commented 2 weeks ago

related fix https://github.com/apache/seatunnel/pull/4469#issuecomment-2372833258, but it not update long time. @fcb-xiaobo hi, can you help fix this issue?

Carl-Zhou-CN commented 2 weeks ago

I also looked into it and provided some information, it should be that write and commit both hold the same client.id producer, conflict

fcb-xiaobo commented 2 weeks ago

I am currently trying to reproduce this issue,and I will try to fix it later

corgy-w commented 2 weeks ago

related fix #4469 (comment), but it not update long time. @fcb-xiaobo hi, can you help fix this issue?

@fcb-xiaobo As long as you configure 'semantics = EXACTLY_ONCE' to create this problem, pr referenced by @liunaijie is a good idea but needs some optimization,I briefly tried it

corgy-w commented 3 days ago

I am currently trying to reproduce this issue,and I will try to fix it later

@fcb-xiaobo Hi, is there any progress here?

fcb-xiaobo commented 2 days ago

I am currently trying to reproduce this issue,and I will try to fix it later

@fcb-xiaobo Hi, is there any progress here?

The local code has been modified, but there are still issues with the e2e test case running on Docker, which I am still working on resolving