lolaclinton
commented
6 years ago The session is initialized just as in your example
val sc = new SparkContext(new SparkConf())Open lolaclinton opened 6 years ago
lolaclinton
commented
6 years ago The session is initialized just as in your example
val sc = new SparkContext(new SparkConf())
pishen
commented
6 years ago @lolaclinton How did you submit the job? Where did you see this log?
HI :), Sorry to bug you again. Let me know if you can help. So I'm running a Spark job with your plugin. It runs fine on EMR with an 8 machine cluster for a couple hours all of a sudden it terminates with this strange error:
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] {"ts":"2017-10-11T21:57:09.275+00:00","msg":"Error initializing SparkContext.","logger":"org.apache.spark.SparkContext","level":"ERROR","stack_trace":"org.apache.spark.SparkException: A master URL must be set in your configuration\n\tat org.apache.spark.SparkContext.(SparkContext.scala:379) ~[spark-core_2.11-2.1.0.jar:2.1.0]\n\tat RunExtractors$.main(RunExtractors.scala:28) [classes/:na]\n\tat RunExtractors.main(RunExtractors.scala) [classes/:na]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_66]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_66]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]\n\tat java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]\n\tat sbt.Run.invokeMain(Run.scala:67) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run.run0(Run.scala:61) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run.sbt$Run$$execute$1(Run.scala:51) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run$$anonfun$run$1.apply(Run.scala:55) [run-0.13.13.jar:0.13.13]\n\tat sbt.Run$$anonfun$run$1.apply(Run.scala:55) [run-0.13.13.jar:0.13.13]\n\tat sbt.Logger$$anon$4.apply(Logger.scala:84) [logging-0.13.13.jar:0.13.13]\n\tat sbt.TrapExit$App.run(TrapExit.scala:248) [run-0.13.13.jar:0.13.13]\n\tat java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]\n","HOSTNAME":"Administrators-MacBook-Pro-2.local"}
[error] (run-main-0) org.apache.spark.SparkException: A master URL must be set in your configuration
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.(SparkContext.scala:379)
at RunExtractors$.main(RunExtractors.scala:28)
at RunExtractors.main(RunExtractors.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
any idea?