search

holdenk / spark-flowchart

Flowchart for debugging Spark applications
100 stars 27 forks source link

Documentation idea: how long is my job stuck in accepted state #14

Closed atbida closed 2 years ago

atbida commented 2 years ago

start here --> slow job --> i think my job is accepted by the spark cluster scheduler, but I don't have visibility into the time spent as "Accepted" --> in spark 3.4 we can find this info in the job's JMX metrics

--> Why is my job waiting in accepted state? --> How do I reduce the time my job is waiting in accepted state? --> How do I monitor how long each job is spendng in each (eg accepted) state?

Reference link: https://issues.apache.org/jira/browse/SPARK-36664

atbida commented 2 years ago

Updated this fix from PR18 to PR26 https://github.com/holdenk/spark-flowchart/pull/26

Closing this issue