search

uber-common / jvm-profiler

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter
Other
1.79k stars 342 forks source link

java.lang.NoSuchMethodError: okio.ByteString.encodeString with Spark/InfluxDB #30

Closed yeenow123 closed 4 years ago

yeenow123 commented 5 years ago

When using the InfluxDB reporter on Spark 2.2.0 with Ambari HDP, I'm getting the following error.

I ran mvn -P influxdb clean package to build the jar.

Using the following command (with classname / host replaced):

spark-submit --master yarn-cluster --class com.output.spark.FilteringJob --conf spark.jars=hdfs:///user/smorgasborg/lib/jvm-profiler-1.0.0.jar --conf spark.executor.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.InfluxDBOutputReporter,tag=profiling,sampleInterval=1000,influxdb.host={{HOST HERE}},influxdb.port=8086,influxdb.database=test_profiling --conf spark.yarn.am.waitTime=200s spark-output-assembly.jar

[WARNING] 1544111654753 com.uber.profiling.ProfilerRunner: Failed to run profile: com.uber.profiling.profilers.StacktraceReporterProfiler@b9afc07 java.lang.NoSuchMethodError: okio.ByteString.encodeString(Ljava/lang/String;Ljava/nio/charset/Charset;)Lokio/ByteString;
    at okhttp3.Credentials.basic(Credentials.java:35)
    at okhttp3.Credentials.basic(Credentials.java:30)
    at ujagent_shaded.org.influxdb.impl.BasicAuthInterceptor.<init>(BasicAuthInterceptor.java:15)
    at ujagent_shaded.org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:153)
    at ujagent_shaded.org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:122)
    at ujagent_shaded.org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:185)
    at ujagent_shaded.org.influxdb.InfluxDBFactory.connect(InfluxDBFactory.java:48)
    at com.uber.profiling.reporters.InfluxDBOutputReporter.ensureInfluxDBCon(InfluxDBOutputReporter.java:148)
    at com.uber.profiling.reporters.InfluxDBOutputReporter.report(InfluxDBOutputReporter.java:57)
    at com.uber.profiling.profilers.StacktraceReporterProfiler.profile(StacktraceReporterProfiler.java:118)
    at com.uber.profiling.ProfilerRunner.run(ProfilerRunner.java:38)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
yeenow123 commented 5 years ago

Created this pull request, okio version in the influxdb dependency conflicted with the Spark okio version

https://github.com/uber-common/jvm-profiler/pull/31

baghelamit commented 5 years ago

Hi @yeenow123, I couldn't reproduce this error with Spark-2.2.0. I checked the spark-deps-hadoop-2.7 for spark branch-2.2 but this branch doesn't show any dependency for okhttp and okio jar files. branch-2.3 and branch-2.4 listed the dependency for okhttp and okio jar files and versions given are compatible with InfluxDBOutputReporter. It looks like there is some dependency in your spark application jar which is bringing the older version of okhttp and okio jar files. Could you please check the dependency tree for your spark application and identify that jar file? Please also check the version of okhttp and okio jar files in that dependency tree. This will help us to narrow down the issue and applying the fix. Thanks.

c-horn commented 5 years ago

I am also seeing this error with spark 2.3

I am bringing in 1.14 okio via --repositories:

[info]   | | +-org.influxdb:influxdb-java:2.14
[info]   | |   +-com.squareup.okhttp3:logging-interceptor:3.11.0
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   |   +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   | +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-com.squareup.retrofit2:converter-moshi:2.4.0
[info]   | |   | +-com.squareup.moshi:moshi:1.5.0
[info]   | |   | | +-com.squareup.okio:okio:1.13.0 (evicted by: 1.14.0)
[info]   | |   | | +-com.squareup.okio:okio:1.14.0
[info]   | |   | |
[info]   | |   | +-com.squareup.retrofit2:retrofit:2.4.0
[info]   | |   |   +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   |   +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   |     +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-com.squareup.retrofit2:retrofit:2.4.0
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   |   +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-org.msgpack:msgpack-core:0.8.16

Nothing else on my dependency list pulls in any square/okio/okhttp.

c-horn commented 5 years ago

It appears this only happens in cluster mode, with client mode I do not get the error running an identical spark job.

The cluster mode submit is including this jar on the system classpath: /usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar

EDIT: correction, in client mode it appears the driver does not receive the jar in question, but the executors still do.

felixcheung commented 5 years ago

in client mode, your driver is the spark submit process...

it looks like /usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar is injected by YARN?

g1thubhub commented 5 years ago

Hey @felixcheung and @yeenow123, this issue might also be caused by the relocations in the POM file -- I had a similar "class not found issue" when testing a new OutputReporter that I'll commit soon, see this PR for how I fixed it the issue: https://github.com/uber-common/jvm-profiler/pull/51

hanhtd2 commented 4 years ago

Actually in my case, I fixed it. I used maven and change the version of influxdb: Try this:

org.influxdb influxdb-java 2.7
hiboyang commented 4 years ago

Thanks hanhtd2 for the information! I updated influxdb-java to 2.7 in pom.xml just now.