search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.77k stars 4.59k forks source link

[Bug] [task-plugin] run spark application failed #7794

Closed QlDoors closed 2 years ago

QlDoors commented 2 years ago

Search before asking

What happened

when I run spark applition on yarn, I got a failed messasge: 22/01/04 19:24:27 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804) at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795) at org.example.WordCount$.main(WordCount.scala:40) at org.example.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567) 22/01/04 19:24:27 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;) 22/01/04 19:24:27 ERROR yarn.ApplicationMaster: Uncaught exception: java.util.concurrent.ExecutionException: Boxed Error at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244) at scala.concurrent.Promise$class.tryFailure(Promise.scala:112) at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:585) Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804) at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795) at org.example.WordCount$.main(WordCount.scala:40) at org.example.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567) 22/01/04 19:24:27 INFO util.ShutdownHookManager: Shutdown hook called

What you expected to happen

run successs

How to reproduce

  1. dolphinscheduler version 2.0.1，download bin.jar from https://dolphinscheduler.apache.org/
  2. config conf/env/dolphinscheduler_env.sh 
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf
#export SPARK_HOME1=/opt/cloudera/parcels/CDH/lib/spark
export SPARK_HOME2=/opt/cloudera/parcels/SPARK2/lib/spark2
export PYTHON_HOME=/usr/bin/python
export JAVA_HOME=/usr/java/jdk1.8.0_161
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
#export FLINK_HOME=/opt/soft/flink
#export DATAX_HOME=/opt/soft/datax

export PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH

  1. cdh version 5.14.4，spark1 version 1.6.0，spark2 version 2.4.0.cloudera2

  2. put /opt/cloudera/parcels/SPARK2/lib/spark2/jars/ *.jar to /dolphinscheduler/qinliang/resources/spark-jars/

  3. work-flow 'Option Parameters':

    --jars hdfs:///dolphinscheduler/qinliang/resources/spark-jars/JavaEWAH-0.3.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/RoaringBitmap-0.7.45.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/ST4-4.0.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/aircompressor-0.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/antlr-2.7.7.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/antlr-runtime-3.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/antlr4-runtime-4.7.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/aopalliance-repackaged-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/apache-log4j-extras-1.2.17.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arpack_combined_all-0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arrow-format-0.10.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arrow-memory-0.10.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/arrow-vector-0.10.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/avro-1.7.6-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/avro-ipc-1.7.6-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/avro-mapred-1.7.6-cdh5.13.3-hadoop2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/bonecp-0.7.1.RELEASE.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/breeze-macros_2.11-0.13.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/breeze_2.11-0.13.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/calcite-avatica-1.0.0-incubating.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/calcite-core-1.0.0-incubating.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/calcite-linq4j-1.0.0-incubating.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/chill-java-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/chill_2.11-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-cli-1.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-codec-1.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-collections-3.2.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-compiler-3.0.9.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-compress-1.4.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-crypto-1.0.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-dbcp-1.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-httpclient-3.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-io-2.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-lang-2.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-lang3-3.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-logging-1.1.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-math3-3.4.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-net-3.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/commons-pool-1.5.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/compress-lzf-1.0.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/core-1.1.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/datanucleus-api-jdo-3.2.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/datanucleus-core-3.2.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/datanucleus-rdbms-3.2.9.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/derby-10.12.1.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/eigenbase-properties-1.1.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/fastutil-6.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flatbuffers-1.2.0-3f79e055.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flume-ng-configuration-1.6.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flume-ng-core-1.6.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/flume-ng-sdk-1.6.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/gson-2.2.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-metastore-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-serde-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-0.23-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-common-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hive-shims-scheduler-1.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hk2-api-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hk2-locator-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hk2-utils-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/hppc-0.7.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/httpcore-4.4.10.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/ivy-2.4.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-annotations-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-core-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-databind-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-jaxrs-1.8.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-module-paranamer-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-module-scala_2.11-2.9.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jackson-xc-1.8.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/janino-3.0.9.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javassist-3.18.1-GA.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.annotation-api-1.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.inject-2.4.0-b34.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.servlet-api-3.1.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javax.ws.rs-api-2.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/javolution-5.5.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jcl-over-slf4j-1.7.25.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jdo-api-3.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-client-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-common-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-container-servlet-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-container-servlet-core-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-guava-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-media-jaxb-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jersey-server-2.22.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jetty-6.1.26.cloudera.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jetty-util-6.1.26.cloudera.4.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jline-2.14.6.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/joda-time-2.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jodd-core-3.5.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json-20090211.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-ast_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-core_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-jackson_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/json4s-scalap_2.11-3.5.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jta-1.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jtransforms-2.4.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/jul-to-slf4j-1.7.25.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/kryo-shaded-4.0.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/libfb303-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/libthrift-0.9.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/logredactor-1.0.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/lz4-java-1.4.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/machinist_2.11-0.6.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/macro-compat_2.11-1.1.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-core-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-graphite-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-json-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/metrics-jvm-3.1.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/mina-core-2.0.0-M5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/minlog-1.3.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/netty-3.9.9.Final.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/netty-all-4.1.17.Final.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/objenesis-2.5.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/opencsv-2.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/orc-core-1.5.5-nohive.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/orc-mapreduce-1.5.5-nohive.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/orc-shims-1.5.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/oro-2.0.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/osgi-resource-locator-1.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/paranamer-2.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-avro-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-column-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-common-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-encoding-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-format-2.1.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-hadoop-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-hadoop-bundle-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/parquet-jackson-1.5.0-cdh5.13.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/py4j-0.10.7.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/pyrolite-4.13.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-compiler-2.11.12.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-library-2.11.12.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-parser-combinators_2.11-1.1.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-reflect-2.11.12.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/scala-xml_2.11-1.0.5.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/shapeless_2.11-2.3.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/shims-0.7.45.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/snappy-0.2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/snappy-java-1.1.7.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-catalyst_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-core_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-graphx_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-hive-exec_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-hive_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-kvstore_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-launcher_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-lineage_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-mllib-local_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-mllib_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-network-common_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-network-shuffle_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-repl_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-sketch_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-sql_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-streaming-flume-sink_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-streaming-flume_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-streaming_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-tags_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-unsafe_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spark-yarn_2.11-2.4.0.cloudera2.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spire-macros_2.11-0.13.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/spire_2.11-0.13.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/stax-api-1.0.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/stream-2.7.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/stringtemplate-3.2.1.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/univocity-parsers-2.7.3.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/validation-api-1.1.0.Final.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/xbean-asm6-shaded-4.8.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/xz-1.0.jar,hdfs:///dolphinscheduler/qinliang/resources/spark-jars/zstd-jni-1.3.2-2.jar

  4. run work-flow，yarn logs：

    22/01/04 19:24:27 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795)
at org.example.WordCount$.main(WordCount.scala:40)
at org.example.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
22/01/04 19:24:27 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
22/01/04 19:24:27 ERROR yarn.ApplicationMaster: Uncaught exception: 
java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:585)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:804)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:795)
at org.example.WordCount$.main(WordCount.scala:40)
at org.example.WordCount.main(WordCount.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
22/01/04 19:24:27 INFO util.ShutdownHookManager: Shutdown hook called

Anything else

No response

Version

2.0.1

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 2 years ago

Hi:

QlDoors commented 2 years ago

解决了，因为dolphinscheduler在执行命令的时候，没有把 conf/env/dolphinscheduler_env.sh 加载到运行环境中造成的，通过创建一个"环境"搞定了，环境配置如下：

source /etc/profile
source /home/qinliang/dolphinscheduler/conf/env/dolphinscheduler_env.sh
export HADOOP_USER_NAME=hdfs