search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

How to get the execution time for EACH operator in Spark SQL? #35

Closed niewangwen closed 2 years ago

niewangwen commented 2 years ago

Execuse me, I'm having some problem when getting the execution time of each operator in Spark SQL. For example, I'm using the belowed sql to test:
"select count(*) from range(1000) cross join range(1000) cross join range(1000)" and I found some details in the Spark Web UI like this: 4 I wish to get the execution time of every operator, but operators like "RANGE, PROJECT etc" doesn't have the corresponding metrics. I look into the log and I find that those operator doesn't have any metrics, so maybe this is the reason. But I want to get execution time for each operator, how can I do that?

LucaCanali commented 2 years ago

Hi, I agree that would be useful, but I am afraid Spark does not instrument each operator with time.