search

qubole / sparklens

Qubole Sparklens tool for performance tuning Apache Spark
http://sparklens.qubole.com
Apache License 2.0
567 stars 138 forks source link

support in spark 3.x version #64

Open abdul2011 opened 3 years ago

abdul2011 commented 3 years ago

Does sparklens supported with spark 3.x version

utkarsh92 commented 2 years ago

i need to know this as well. i tried running a simple WordCount application with spark-3.2.1 and kubernetes-1.20 cluster setup. i added the lines --jars /path/to/sparklens-0.3.2-s_2.11.jar --conf spark.extraListeners=com.qubole.sparklens.QuboleJobListener in my spark-submit query. the result was that the executor pods keep on running INFINITELY. they never stop and always stay in "Running" status.

link to kubectl logs

Update: this bug seems to happen with Spark-Pi example as well :\