search

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.45k stars 2.43k forks source link

[SUPPORT] Apply 'write.ignore.failed' when write data failed #12187

Open fhan688 opened 3 weeks ago

fhan688 commented 3 weeks ago

Describe the problem you faced

In Flink engine, if exception occurs when task writing data, it will be ignored and the exception will be reported to StreamWriteCoordinator with write event, StreamWriteCoordinator will decide whether to commit when there is write failure according to 'write.ignore.failed'.

Whether we should apply 'write.ignore.failed' ahead when write failure occurs, thus throw an exception faster.

To Reproduce

Expected behavior

  1. promote 'write.ignore.failed' to hudi-client-common
  2. throw exception in write handles
  3. default value of 'write.ignore.failed' is false

Environment Description

Additional context

related PR：https://github.com/apache/hudi/pull/12150

Stacktrace

danny0405 commented 3 weeks ago

@nbalajee Do you have some insights about promote the 'write.ignore.failed' as a basic option for all the engines? I kind of remember we had a discussion before and engine specific maintainance is prefferable.