search

qubole / sparklens

Qubole Sparklens tool for performance tuning Apache Spark
http://sparklens.qubole.com
Apache License 2.0
567 stars 138 forks source link

Sparklens with spark-submit #7

Closed akumarb2010 closed 6 years ago

akumarb2010 commented 6 years ago

Once we run the spark application with spark-submit by adding below sparklens configs, where can we find the sparklens results? Or does Sparklens only works with notebooks?

--packages qubole:sparklens:0.1.2-s_2.11
--conf spark.extraListeners=com.qubole.sparklens.QuboleJobListener
iamrohit commented 6 years ago

Check the driver logs.

akumarb2010 commented 6 years ago

Thanks for quick response. I am able to see these reports logged into Driver logs.

Just for quick check, is there any web interface for accessing reports as mentioned in https://www.qubole.com/blog/introducing-quboles-spark-tuning-tool/ ? Just wanted to understand critical path for our use cases.

iamrohit commented 6 years ago

No we don't yet have any UI to view this. We do print the length (time taken) in the critical path. Look for text like: Minimum possible time for the app based on the critical path (with infinite resources)

akumarb2010 commented 6 years ago

Thanks for the details @iamrohit . It's very helpful.