Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
26 stars 5 forks source link

Reduce logging #898

Closed EmileSonneveld closed 5 days ago

jdries commented 1 month ago

As it happens, I just extended task failure handling in the listener to suppress specific warnings. Was the logging of TaskSetManager somehow better than the listener?

EmileSonneveld commented 1 month ago

TaskSetManager would log failing tasks as warning, and only make it an error if the last retry fails. It also deduplicates some messages. Source code here: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L968