search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

Memory usage #27

Closed ductuantruong closed 2 years ago

ductuantruong commented 4 years ago

Can we get memory usage of a spark-submit? stagemetrics.print_report() don't have that.

LucaCanali commented 2 years ago

This is fixed in sparkMeasure v0.21 which instroduced executor metrics collection and the reports: 

(scala)> stageMetrics.printMemoryReport
(python)> stagemetrics.print_memory_report()