Closed EmileSonneveld closed 5 days ago
TaskSetManager would log failing tasks as warning, and only make it an error if the last retry fails. It also deduplicates some messages. Source code here: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L968
As it happens, I just extended task failure handling in the listener to suppress specific warnings. Was the logging of TaskSetManager somehow better than the listener?