search

microsoft / azure-tools-for-java

Azure tools for Java, including Azure Toolkits for Eclipse, IntelliJ and related projects.
Other
239 stars 165 forks source link

job failed with ClassNotFoundException when run configuration without mainclass #1324

Closed voidfunction closed 6 years ago

voidfunction commented 6 years ago
  1. Create a new project.
  2. Create a run configuration without main class. And then save the configuration. image
  3. Click the run button without main class. image
  4. Job failed with ClassNotFoundException. 
    LOG: AM container is launched, waiting for AM container to Register with RM
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.5.3-2/spark2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.5.3-2/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/12/29 10:44:00 INFO SignalUtils: Registered signal handler for TERM
17/12/29 10:44:00 INFO SignalUtils: Registered signal handler for HUP
17/12/29 10:44:00 INFO SignalUtils: Registered signal handler for INT
17/12/29 10:44:01 INFO ApplicationMaster: Preparing Local resources
17/12/29 10:44:01 INFO MetricsConfig: loaded properties from hadoop-metrics2-azure-file-system.properties
17/12/29 10:44:01 INFO WasbAzureIaasSink: Init starting.
17/12/29 10:44:01 INFO AzureIaasSink: Init starting. Initializing MdsLogger.
17/12/29 10:44:01 INFO AzureIaasSink: Init completed.
17/12/29 10:44:01 INFO WasbAzureIaasSink: Init completed.
17/12/29 10:44:01 INFO MetricsSinkAdapter: Sink azurefs2 started
17/12/29 10:44:01 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
17/12/29 10:44:01 INFO MetricsSystemImpl: azure-file-system metrics system started
17/12/29 10:44:01 INFO ApplicationMaster: Prepared Local resources Map(__app__.jar -> resource { scheme: "wasb" host: "zhwespk.blob.core.windows.net" port: -1 file: "/SparkSubmission/2017/12/29/ac19901e-acbf-4a30-97a5-c4063192e1ce/default_artifact.jar" userInfo: "zhwe-spkdbg2" } size: 67484 timestamp: 1514544225000 type: FILE visibility: PUBLIC, __spark_conf__ -> resource { scheme: "wasb" host: "zhwespk.blob.core.windows.net" port: -1 file: "/user/livy/.sparkStaging/application_1511284610088_0199/__spark_conf__.zip" userInfo: "zhwe-spkdbg2" } size: 108659 timestamp: 1514544229000 type: ARCHIVE visibility: PRIVATE)
17/12/29 10:44:01 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1511284610088_0199_000004
17/12/29 10:44:01 INFO SecurityManager: Changing view acls to: yarn,livy
17/12/29 10:44:01 INFO SecurityManager: Changing modify acls to: yarn,livy
17/12/29 10:44:01 INFO SecurityManager: Changing view acls groups to: 
17/12/29 10:44:01 INFO SecurityManager: Changing modify acls groups to: 
17/12/29 10:44:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, livy); groups with view permissions: Set(); users  with modify permissions: Set(yarn, livy); groups with modify permissions: Set()
17/12/29 10:44:01 INFO ApplicationMaster: Starting the user application in a separate Thread
17/12/29 10:44:01 ERROR ApplicationMaster: Uncaught exception: 
java.lang.ClassNotFoundException: 
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:621)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:379)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:749)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:747)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
17/12/29 10:44:01 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.ClassNotFoundException: )
17/12/29 10:44:01 INFO ShutdownHookManager: Shutdown hook called
17/12/29 10:44:01 INFO MetricsSystemImpl: Stopping azure-file-system metrics system...
17/12/29 10:44:01 INFO MetricsSinkAdapter: azurefs2 thread interrupted.
17/12/29 10:44:01 INFO MetricsSystemImpl: azure-file-system metrics system stopped.
17/12/29 10:44:01 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.5.3-2/spark2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.5.3-2/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/12/29 10:44:03 INFO SignalUtils: Registered signal handler for TERM
17/12/29 10:44:03 INFO SignalUtils: Registered signal handler for HUP
17/12/29 10:44:03 INFO SignalUtils: Registered signal handler for INT
17/12/29 10:44:03 INFO ApplicationMaster: Preparing Local resources
17/12/29 10:44:04 INFO MetricsConfig: loaded properties from hadoop-metrics2-azure-file-system.properties
17/12/29 10:44:04 INFO WasbAzureIaasSink: Init starting.
17/12/29 10:44:04 INFO AzureIaasSink: Init starting. Initializing MdsLogger.
17/12/29 10:44:04 INFO AzureIaasSink: Init completed.
17/12/29 10:44:04 INFO WasbAzureIaasSink: Init completed.
17/12/29 10:44:04 INFO MetricsSinkAdapter: Sink azurefs2 started
17/12/29 10:44:04 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
17/12/29 10:44:04 INFO MetricsSystemImpl: azure-file-system metrics system started
17/12/29 10:44:04 INFO ApplicationMaster: Prepared Local resources Map(__app__.jar -> resource { scheme: "wasb" host: "zhwespk.blob.core.windows.net" port: -1 file: "/SparkSubmission/2017/12/29/ac19901e-acbf-4a30-97a5-c4063192e1ce/default_artifact.jar" userInfo: "zhwe-spkdbg2" } size: 67484 timestamp: 1514544225000 type: FILE visibility: PUBLIC, __spark_conf__ -> resource { scheme: "wasb" host: "zhwespk.blob.core.windows.net" port: -1 file: "/user/livy/.sparkStaging/application_1511284610088_0199/__spark_conf__.zip" userInfo: "zhwe-spkdbg2" } size: 108659 timestamp: 1514544229000 type: ARCHIVE visibility: PRIVATE)
17/12/29 10:44:04 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1511284610088_0199_000005
17/12/29 10:44:04 INFO SecurityManager: Changing view acls to: yarn,livy
17/12/29 10:44:04 INFO SecurityManager: Changing modify acls to: yarn,livy
17/12/29 10:44:04 INFO SecurityManager: Changing view acls groups to: 
17/12/29 10:44:04 INFO SecurityManager: Changing modify acls groups to: 
17/12/29 10:44:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, livy); groups with view permissions: Set(); users  with modify permissions: Set(yarn, livy); groups with modify permissions: Set()
17/12/29 10:44:04 INFO ApplicationMaster: Starting the user application in a separate Thread
17/12/29 10:44:04 ERROR ApplicationMaster: Uncaught exception: 
java.lang.ClassNotFoundException: 
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:621)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:379)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:749)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:747)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

I think we should disable the run button if configuration verification failed and give some tips for customers.

jingyanjingyan commented 6 years ago

warning message display after starting remote debug.

HuyAnh commented 5 years ago

I have same the error, please help me

wezhang commented 5 years ago

@HuyAnh , you might try to fill Main class name field before submitting since the Spark cluster needs a main class entry point for your application, just like a normal Java application.