search

uber-common / jvm-profiler

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter
Other
1.78k stars 342 forks source link

a bug has appeared when I use jvm-profiler with spark client mode #64

Open CCweixiao opened 4 years ago

CCweixiao commented 4 years ago

code

spark-submit  \
 --master yarn  \
 --deploy-mode client  \
 --executor-memory 10G  \
 --executor-cores 3  \
 --driver-memory 5g   \
  --conf "spark.driver.extraJavaOptions=-javaagent:/data/leo_jie/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.InfluxDBOutputReporter,tag=influxdb,configProvider=com.uber.profiling.YamlConfigProvider,configFile=/data/leo_jie/Influxdb.yaml,metricInterval=5000,sampleInterval=5000,ioProfiling=true" \
 --files /data/leo_jie/Influxdb.yaml \
 --jars /data/leo_jie/jvm-profiler-1.0.0.jar \
 /data/leo_jie/spark_test.py

quistion

The spark application won't exit automatically when the application finished!

image

image

hiboyang commented 4 years ago

Thanks for reporting the issue. We haven't tried the profiler with spark client mode yet. You are welcome to submit a fix if you are interested to solve this issue :)

yanivyalda commented 4 years ago

hey @CCweixiao im having the same issue today with the influxdb reporter. It only happens with that reporter. I've tried console and file output and they dont have this issue.