This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690
stars
144
forks
source link
How to get the execution time for EACH operator in Spark SQL? #35
Execuse me, I'm having some problem when getting the execution time of each operator in Spark SQL. For example, I'm using the belowed sql to test:
"select count(*) from range(1000) cross join range(1000) cross join range(1000)"
and I found some details in the Spark Web UI like this:
I wish to get the execution time of every operator, but operators like "RANGE, PROJECT etc" doesn't have the corresponding metrics.
I look into the log and I find that those operator doesn't have any metrics, so maybe this is the reason.
But I want to get execution time for each operator, how can I do that?
Execuse me, I'm having some problem when getting the execution time of each operator in Spark SQL. For example, I'm using the belowed sql to test:
I wish to get the execution time of every operator, but operators like "RANGE, PROJECT etc" doesn't have the corresponding metrics.
I look into the log and I find that those operator doesn't have any metrics, so maybe this is the reason.
But I want to get execution time for each operator, how can I do that?
"select count(*) from range(1000) cross join range(1000) cross join range(1000)" and I found some details in the Spark Web UI like this: