search

qubole / sparklens

Qubole Sparklens tool for performance tuning Apache Spark
http://sparklens.qubole.com
Apache License 2.0
567 stars 138 forks source link

Report driver memory statistics with QuboleJobListener #37

Open mayurdb opened 5 years ago

mayurdb commented 5 years ago

Following metrics related to the Driver are now reported -

  1. driverHeapMax => Max Heap memory allocated to the driver JVM
  2. driverMaxHeapCommitted => Max Heap memory committed to the driver JVM
  3. driverMaxHeapUsed => Max Heap memory used by the driver JVM
  4. driverCPUTime => Total CPU Time allocated to the driver JVM
  5. driverGCCount => Total major GCs happened during the lifetime of the JVM
  6. driverGCTime => Part of driverCPUTime spent in the GC

Sample output for a simple application.


 Time spent in Driver vs Executors
 Driver WallClock Time    00m 11s   79.24%
 Executor WallClock Time  00m 03s   20.76%
 Total WallClock Time     00m 15s

 DriverMetrics (driverMetrics)
                NAME                        Value
 driverCPUTime                           10216.7 hh
 driverGCCount                                   11
 driverGCTime                              301.0 ms
 driverHeapMax                               1.0 GB
 driverMaxHeapCommitted                    480.0 MB
 driverMaxHeapUsed                         308.9 MB