search

uber-common / jvm-profiler

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter
Other
1.78k stars 342 forks source link

Give more details on how to use this with Spark Application #15

Closed anujchoudhury closed 6 years ago

anujchoudhury commented 6 years ago

Could more details be given on how to use the Profiler (with different Consoles) for a spark application ? I could find the details in the Readme for a normal Java Application, but the details on how to use it with a spark application could not be found.

petro-rudenko commented 6 years ago

Just add: spark.executor.extraJavaOptions -Djava.library.path=/jvm-profiler/target/jvm-profiler-0.0.7.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=mytag,metricInterval=5000,sampleInterval=100

More details here: https://eng.uber.com/jvm-profiler/

You'll see results in stdout of yarn logs. But would be great to aggregate metrics and visualize them in some way, e.g like in : https://github.com/criteo/babar

hiboyang commented 6 years ago

Thanks Peter for replying! I will close this issue, and update README with link to https://eng.uber.com/jvm-profiler/ as well.

Also it is great that you mentioned visualization and the example for babar, we haven't got time to work on that yet.