judynash
commented
8 years ago Adding comments to explain why this is happening. This is a Spark by-design issue. When a spark application is launched, it scans for an available port to use, starting from 4040 to 4041, 4042, 4043, etc. For HDInsight spark cluster, 4040 is generally taken by spark thrift server, so the first available server is 4041. That's why the application sees this at launch: java.net.BindException: Address already in use
This will go away on its own once app finds an available, but alarming to users nonetheless.
Although Spark 2.0.0 is working, it's still giving me exceptions:
sshuser@hn0-tlcspa:~$ spark-shellWarning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.4-5/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.4-5/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 15:54:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/08/26 15:54:35 WARN AbstractLifeCycle: FAILED ServerConnector@4dcc1f71{HTTP/1.1}{0.0.0.0:4040}: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:321) at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:236) at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.spark_project.jetty.server.Server.doStart(Server.java:366) at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:298) at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:308) at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:308) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2071) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2062) at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:308) at org.apache.spark.ui.WebUI.bind(WebUI.scala:139) at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:451) at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:451) at scala.Option.foreach(Option.scala:257) at org.apache.spark.SparkContext.// ./,// //_\ version 2.0.0 //
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.7.0_101) Type in expressions to have them evaluated. Type :help for more information.
scala>