search

qubole / sparklens

Qubole Sparklens tool for performance tuning Apache Spark
http://sparklens.qubole.com
Apache License 2.0
567 stars 138 forks source link

analysize spark eventhistory , but per stage metrics, max task mem is all 0.0KB #77

Open ghost opened 3 years ago

ghost commented 3 years ago

sparklens-0.3.2-s_2.11.jar spark version: 2.4.0 why the column maxTaskMem is 0.0KB, i try 2 different tasks, all shows like below: image

Huyng19 commented 2 years ago

I think they already have an answer here: https://github.com/qubole/sparklens/issues/21 . Sounds like sparklens doesn't work with event logs mode properly for this case due to Spark's JsonProtocol class