search

qubole / sparklens

Qubole Sparklens tool for performance tuning Apache Spark
http://sparklens.qubole.com
Apache License 2.0
567 stars 138 forks source link

Not able to run spark lens on spark history file #61

Open Gaurang033 opened 4 years ago

Gaurang033 commented 4 years ago

hi Guys,

I am trying to get spark lens report from my history file. However it's not able to identify the arguments.

I am getting following error 

[sa_awsprd_nztoaws@ip-10-160-1-109 gshah]$ spark-submit --jars sparklens-0.3.2-s_2.11.jar  --class com.qubole.sparklens.app.ReporterApp qubole-dummy-arg  file:////home/gshah/application_1589573238486_1924 source=file
20/09/03 14:57:22 WARN DependencyUtils: Local jar /home/gshah/qubole-dummy-arg does not exist, skipping.
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 3788"...
Killed
iamrohit commented 4 years ago

@Gaurang033 looks like you are running our of driver heap. Try increasing driver memory: spark-submit --driver-memory 4g --jars sparklens-0.3.2-s_2.11.jar --class com.qubole.sparklens.app.ReporterApp qubole-dummy-arg file:////home/gshah/application_1589573238486_1924 source=file