search

pishen / sbt-lighter

SBT plugin for Apache Spark on AWS EMR
Apache License 2.0
57 stars 15 forks source link

another stupid question #21

Open lolaclinton opened 7 years ago

lolaclinton commented 7 years ago

HI :), Sorry to bug you again. Let me know if you can help. So I'm running a Spark job with your plugin. It runs fine on EMR with an 8 machine cluster for a couple hours all of a sudden it terminates with this strange error:

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] {"ts":"2017-10-11T21:57:09.275+00:00","msg":"Error initializing SparkContext.","logger":"org.apache.spark.SparkContext","level":"ERROR","stack_trace":"org.apache.spark.SparkException: A master URL must be set in your configuration\n\tat org.apache.spark.SparkContext.(SparkContext.scala:379) ~[spark-core_2.11-2.1.0.jar:2.1.0]\n\tat RunExtractors$.main(RunExtractors.scala:28) [classes/:na]\n\tat RunExtractors.main(RunExtractors.scala) [classes/:na]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_66]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_66]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]\n\tat java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]\n\tat sbt.Run.invokeMain(Run.scala:67) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run.run0(Run.scala:61) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run.sbt$Run$$execute$1(Run.scala:51) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run$$anonfun$run$1.apply(Run.scala:55) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run$$anonfun$run$1.apply(Run.scala:55) [run-0.13.13.jar:0.13.13]\n\tat sbt.Logger$$anon$4.apply(Logger.scala:84) [logging-0.13.13.jar:0.13.13]\n\tat sbt.TrapExit$App.run(TrapExit.scala:248) [run-0.13.13.jar:0.13.13]\n\tat java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]\n","HOSTNAME":"Administrators-MacBook-Pro-2.local"} [error] (run-main-0) org.apache.spark.SparkException: A master URL must be set in your configuration org.apache.spark.SparkException: A master URL must be set in your configuration at org.apache.spark.SparkContext.(SparkContext.scala:379) at RunExtractors$.main(RunExtractors.scala:28) at RunExtractors.main(RunExtractors.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497)

any idea?

lolaclinton commented 7 years ago

The session is initialized just as in your example

val sc = new SparkContext(new SparkConf())
pishen commented 7 years ago

@lolaclinton How did you submit the job? Where did you see this log?