Closed ghukill closed 6 years ago
When viewing a Combine Job that is running, perhaps create a tab showing these stages in detail, while Jobs that are complete, default to the Records tab?
More on setting jobGroup
information: https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/SparkContext.html#setJobGroup(java.lang.String,%20java.lang.String,%20boolean)
Completed as "Spark Details" tab in Job Details
Currently, users only know if a Job is waiting, running, or complete. But the Spark API provides some additional information that could be used to inform users of what is happening.
Sample output from API for an entire Combine Job, which is grouped in Spark as a
jobGroup
, each with individualjobs
:Because Spark evaluates lazily, it's hard to estimate how long a Job will take, or even how many stages. But it's possible to name the various Spark jobs, and provide stages / completed for the user, to at least see that the Job is running.