Currently we use ValueOrDie to check the status, such as LazyBatchIterator.AdvanceTo, and it will directly hang up the current process if there is an exception caused bad status. However, in many cases, it is sufficient to just throw an exception rather than hang up the process, such as the FetchFailedException, which means fetch block failed from other executor. And for Spark, the hanging of the executor process is much more serious than throwing exception, because this will lead to a large number of task recalculations, which may lead to more serious problems.
github-actions[bot]
commented
1 year ago
What changes were proposed in this pull request?
Currently we use ValueOrDie to check the status, such as LazyBatchIterator.AdvanceTo, and it will directly hang up the current process if there is an exception caused bad status. However, in many cases, it is sufficient to just throw an exception rather than hang up the process, such as the FetchFailedException, which means fetch block failed from other executor. And for Spark, the hanging of the executor process is much more serious than throwing exception, because this will lead to a large number of task recalculations, which may lead to more serious problems.
How was this patch tested?
unit tests and manual tests.