apache / doris-spark-connector

Spark Connector for Apache Doris
https://doris.apache.org/
Apache License 2.0
74 stars 90 forks source link

[Bug] data loss when enable 2pc #205

Closed zhaorongsheng closed 1 month ago

zhaorongsheng commented 1 month ago

Search before asking

Version

1.3.1

What's Wrong?

The data is lost without any exception when enable doris.sink.enable-2pc and spark jobs running very long time e.g. 3~4hour.

Spark job driver warn logs.

image

The transcation is removed bacause of reaching timeout limit.

image

What You Expected?

It should throw exception when failed to commit transaction .

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct