search

uber-common / jvm-profiler

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter
Other
1.78k stars 342 forks source link

Unable to see JVM Profiler logs #28

Closed harinitya closed 5 years ago

harinitya commented 5 years ago

Hi,

I am trying to run Spark in the client mode, along with the JVM profiler, and I am not able to see any output in the yarn logs. The link below says that the spark.executor.extraJavaOptions does not work in the client mode, even though Spark documentation says its the spark.driver.extraJavaOptions variable that doesn't work.

https://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job

I referred to the link below to see how the values need to be specified

https://github.com/uber-common/jvm-profiler/issues/15

My spark-defaults.conf has the following line

spark.jars=hdfs:///user/lib/jvm-profiler-1.0.0.jar spark.executor.extraJavaOptions -Djava.library.path=/home/spark-user/jvm-prof/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=spark-bench,metricInterval=5000,sampleInterval=100

Does this look ok? Please let me know. And where exactly can I see the profiler output.

Thanks

harinitya commented 5 years ago

Any feedback please? I am not able to use the profiler correctly.

felixcheung commented 5 years ago

did you have javaagent? https://github.com/uber-common/jvm-profiler#example-to-run-with-spark-application

harinitya commented 5 years ago

Thanks! I had seen another post here which asked to use library.path, changing it to -javaagent worked.