Closed Crandel closed 7 years ago
dimaspivak
commented
7 years ago Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
The root user (who you're running as when you start spark-shell) has no user directory in HDFS. If you create one (sudo -u hdfs hdfs dfs -mkdir /user/root followed by sudo -u hdfs dfs -chown root:root /user/root), this should be fixed.
Crandel
commented
7 years ago Thank you very much, it works!
ghost
commented
6 years ago @dimaspivak you are a living legend!
widedh
commented
6 years ago It isn't fixed for me
dimaspivak
commented
6 years ago Paste your full stacktrace, @widedh.
widedh
commented
6 years ago when I try to launch spark-shell I get the same problem, I tried your solution but I doesn't fix it. Thank you
dimaspivak
commented
6 years ago Paste your full stacktrace, @widedh.
widedh
commented
6 years ago Thank you @dimaspivak I fixed the problem by starting hive. I don't understand the link
aman25m
commented
6 years ago I am getting the similar error while running the spark-shell command. Please find the stack trace.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/02/13 22:15:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/13 22:15:48 WARN util.Utils: Your hostname, aman-ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.0.26 instead (on interface wlp2s0)
18/02/13 22:15:48 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/02/13 22:15:50 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (1408+384 MB) is above the max threshold (1536 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:319)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:167)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
at org.apache.spark.SparkContext.
dimaspivak
commented
6 years ago Aman,
What OS/machine configuration are you running on? How much memory, how many CPUs, etc.?
aman25m
commented
6 years ago Hi @dimaspivak I am runnning on Ubuntu 16.04 OS. I have 8 gb ram. Processor: Intel® Core™ i5-7200U CPU @ 2.50GHz × 4.
Getting the following errors: java.lang.IllegalArgumentException: Required executor memory (1408+384 MB) is above the max threshold (1536 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
We have the following details in the configuration files:
File: yarn-site.xml
File: spark-defaults.conf spark.driver.memory 1408m spark.executor.memory 1408m
Changed the configurations to 2048 and now getting the following error: 18/02/13 22:30:43 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 18/02/13 22:31:04 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
Please help
dimaspivak
commented
6 years ago Yeah, you can’t run a full Hadoop cluster with 8 GB of RAM. You really should have 32 GB or so.
aman25m
commented
6 years ago But we are trying to setup a hadoop cluster with 1 master node and 2 data nodes (3 different physical machines ideally). But right now I am trying to work it out on local machine with localhost. Is this the problem?
dimaspivak
commented
6 years ago clusterdock sets up multiple nodes on one machine for testing purposes. You don’t have enough system resources to do what you’re trying to do, which is why it’s failing with messages about insufficient memory.
aman25m
commented
6 years ago Currently we are getting the following error:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/13 22:30:41 WARN util.Utils: Your hostname, aman-ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.0.26 instead (on interface wlp2s0)
18/02/13 22:30:41 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/02/13 22:30:43 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/02/13 22:31:04 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
18/02/13 22:31:04 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:673)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:186)
at org.apache.spark.SparkContext.
vivek2571251
commented
6 years ago Hi, I am getting similar error. Error stack-
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_161)
Type in expressions to have them evaluated.
Type :help for more information.
18/02/14 00:01:15 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
18/02/14 00:01:34 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:123)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.
java.lang.NullPointerException
at org.apache.spark.sql.execution.ui.SQLListener.
MedAzizLass
commented
6 years ago I am having also this issue. What do I need to do?
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/03/09 17:04:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.OutOfMemoryError: Java heap space
at org.apache.spark.executor.Executor$$anonfun$2.apply$mcZI$sp(Executor.scala:91)
at org.apache.spark.executor.Executor$$anonfun$2.apply(Executor.scala:91)
at org.apache.spark.executor.Executor$$anonfun$2.apply(Executor.scala:91)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at org.apache.spark.executor.Executor.
dimaspivak
commented
6 years ago Run it on a machine with more memory.
On Fri, Mar 9, 2018 at 9:05 AM MedAzizLass notifications@github.com wrote:
I am having also this issue. What do I need to do?
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 18/03/09 17:04:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable java.lang.OutOfMemoryError: Java heap space at org.apache.spark.executor.Executor$$anonfun$2.apply$mcZI$sp(Executor.scala:91) at org.apache.spark.executor.Executor$$anonfun$2.apply(Executor.scala:91) at org.apache.spark.executor.Executor$$anonfun$2.apply(Executor.scala:91) at scala.collection.immutable.Range.foreach(Range.scala:160) at org.apache.spark.executor.Executor.(Executor.scala:91) at org.apache.spark.scheduler.local.LocalEndpoint.(LocalSchedulerBackend.scala:59) at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156) at org.apache.spark.SparkContext.(SparkContext.scala:509) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101) ... 17 elided
:14: error: not found: value spark import spark.implicits._ ^ :14: error: not found: value spark import spark.sql ^
Welcome to
/ / ___
/ / \ / / `/ __/ '/ // ._/,// / /_\ version 2.1.1-SNAPSHOT //
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated. Type :help for more information.
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/cloudera/clusterdock/issues/30#issuecomment-371875774, or mute the thread https://github.com/notifications/unsubscribe-auth/AFzozEPTAceimnqDIHWWEfuSGnk8scmtks5tcrZSgaJpZM4Nipy8 .
-- -Dima
snoey
commented
6 years ago I have problem at i run spark-shell
[root@hadoop2 bin]# ./spark-shell 18/03/24 22:14:02 WARN NativeCodeLoader: Unable to load native-hadoop library fo r your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLeve l(newLevel). 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:06 WARN Utils: Service 'sparkDriver' could not bind on a random f ree port. You may check whether configuring an appropriate binding address. 18/03/24 22:15:07 ERROR SparkContext: Error initializing SparkContext. java.net.BindException: Cannot assign requested address: Service 'sparkDriver' f ailed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.dri ver.bindAddress for SparkDriver) to the correct binding address. at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java: 223) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSo cketChannel.java:128) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel. java:558) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChann elPipeline.java:1283) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractCha nnelHandlerContext.java:501) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHa ndlerContext.java:486) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.j ava:989) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364 ) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEv entExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(Single ThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThread EventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorato r.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) java.net.BindException: Cannot assign requested address: Service 'sparkDriver' f ailed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.dri ver.bindAddress for SparkDriver) to the correct binding address. at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketCh annel.java:128) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:5 58) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipe line.java:1283) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHa ndlerContext.java:501) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerC ontext.java:486) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:98 9) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExe cutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThread EventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventE xecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run( DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748)
sunilrdg
commented
5 years ago 19/01/30 12:34:31 WARN General: Plugin (Bundle) "org.datanucleus" is already reg
istered. Ensure you dont have multiple JAR versions of the same plugin in the cl
asspath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already reg
istered, and you are trying to register an identical plugin located at URL "file
:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
19/01/30 12:34:31 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is alr
eady registered. Ensure you dont have multiple JAR versions of the same plugin i
n the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is a
lready registered, and you are trying to register an identical plugin located at
URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
19/01/30 12:34:31 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is
already registered. Ensure you dont have multiple JAR versions of the same plug
in in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is
already registered, and you are trying to register an identical plugin located
at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
19/01/30 12:34:31 WARN Connection: BoneCP specified but not present in CLASSPATH
(or one of dependencies)
19/01/30 12:34:32 WARN Connection: BoneCP specified but not present in CLASSPATH
(or one of dependencies)
19/01/30 12:34:38 WARN ObjectStore: Version information not found in metastore.
hive.metastore.schema.verification is not enabled so recording the schema versio
n 1.2.0
19/01/30 12:34:38 WARN ObjectStore: Failed to get database default, returning No
SuchObjectException
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
a:522)
at org.apache.spark.sql.hive.client.ClientWrapper.
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:10
28)
at $iwC$$iwC.
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.s cala:159) at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkIL oopInit.scala:108) at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala: 64) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark ILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark ILoop$$process$1.apply(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark ILoop$$process$1.apply(SparkILoop.scala:945) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClass Loader.scala:135) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$pr ocess(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSub mit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:18 1) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.NullPointerException at java.lang.ProcessBuilder.start(Unknown Source) at org.apache.hadoop.util.Shell.runCommand(Shell.java:482) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java: 715) at org.apache.hadoop.util.Shell.execCommand(Shell.java:808) at org.apache.hadoop.util.Shell.execCommand(Shell.java:791) at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus. loadPermissionInfo(RawLocalFileSystem.java:582) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus. getPermission(RawLocalFileSystem.java:557) at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(Sess ionState.java:599) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(Sess ionState.java:554) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:508) ... 62 more
vibwipro
commented
4 years ago Hi, When i launch spark shell it dose not invoke SQLContext. How can i invoke SQLcontext on my machine. Please help. I am getting error "error: not found: value sqlContext"
peerfahad
commented
3 years ago Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/11/08 14:39:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x46fa7c39) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x46fa7c39
at org.apache.spark.storage.StorageUtils$.
ainaomotayo
commented
2 years ago @peerfahad I don't think spark-3.2.0 is compatible with Java 17.0.1 right now, try to use other Java version such as Java 8 which works for me.
Dakshmeet
commented
2 years ago can someone help me ??
22/03/23 18:37:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 22/03/23 18:37:39 INFO spark.SecurityManager: Changing view acls to: hadoop 22/03/23 18:37:39 INFO spark.SecurityManager: Changing modify acls to: hadoop 22/03/23 18:37:39 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop) 22/03/23 18:37:39 INFO spark.HttpServer: Starting HTTP Server 22/03/23 18:37:39 INFO server.Server: jetty-8.y.z-SNAPSHOT 22/03/23 18:37:39 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:37459 22/03/23 18:37:39 INFO util.Utils: Successfully started service 'HTTP class server' on port 37459. Welcome to
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/// ./_,// //_\ version 1.6.3 /_/
Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.
22/03/23 18:37:42 WARN util.Utils: Your hostname, dakshmeet-virtual-machine resolves to a loopback address: 127.0.1.1; using 192.168.22.131 instead (on interface ens33)
22/03/23 18:37:42 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
22/03/23 18:37:42 INFO spark.SparkContext: Running Spark version 1.6.3
22/03/23 18:37:42 INFO spark.SecurityManager: Changing view acls to: hadoop
22/03/23 18:37:42 INFO spark.SecurityManager: Changing modify acls to: hadoop
22/03/23 18:37:42 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
22/03/23 18:37:42 INFO util.Utils: Successfully started service 'sparkDriver' on port 34715.
22/03/23 18:37:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
22/03/23 18:37:43 INFO Remoting: Starting remoting
22/03/23 18:37:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.22.131:40555]
22/03/23 18:37:43 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 40555.
22/03/23 18:37:43 INFO spark.SparkEnv: Registering MapOutputTracker
22/03/23 18:37:43 INFO spark.SparkEnv: Registering BlockManagerMaster
22/03/23 18:37:43 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-830328b1-8f3e-4f51-b08b-d237a3c7f8d9
22/03/23 18:37:43 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB
22/03/23 18:37:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
22/03/23 18:37:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
22/03/23 18:37:43 WARN component.AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:461)
at sun.nio.ch.Net.bind(Net.java:453)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)
at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2040)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2031)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.
22/03/23 18:37:49 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=
22/03/23 18:37:49 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_functions: db=default pat=
22/03/23 18:37:49 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
22/03/23 18:37:49 INFO session.SessionState: Created local directory: /tmp/0fe45791-a452-4ac6-898d-4e3387dfa7c8_resources
22/03/23 18:37:49 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/0fe45791-a452-4ac6-898d-4e3387dfa7c8
22/03/23 18:37:49 INFO session.SessionState: Created local directory: /tmp/hadoop/0fe45791-a452-4ac6-898d-4e3387dfa7c8
22/03/23 18:37:49 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/0fe45791-a452-4ac6-898d-4e3387dfa7c8/_tmp_space.db
22/03/23 18:37:49 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse
22/03/23 18:37:49 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
22/03/23 18:37:49 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0
22/03/23 18:37:49 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
22/03/23 18:37:50 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
22/03/23 18:37:50 INFO metastore.ObjectStore: ObjectStore, initialize called
22/03/23 18:37:50 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
22/03/23 18:37:50 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
22/03/23 18:37:50 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
22/03/23 18:37:50 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
22/03/23 18:37:51 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
22/03/23 18:37:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
22/03/23 18:37:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
22/03/23 18:37:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
22/03/23 18:37:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
22/03/23 18:37:52 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
22/03/23 18:37:52 INFO metastore.ObjectStore: Initialized ObjectStore
22/03/23 18:37:53 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
22/03/23 18:37:53 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
22/03/23 18:37:53 INFO metastore.HiveMetaStore: Added admin role in metastore
22/03/23 18:37:53 INFO metastore.HiveMetaStore: Added public role in metastore
22/03/23 18:37:53 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
22/03/23 18:37:53 INFO metastore.HiveMetaStore: 0: get_all_databases
22/03/23 18:37:53 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_all_databases
22/03/23 18:37:53 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=
22/03/23 18:37:53 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_functions: db=default pat=
22/03/23 18:37:53 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
22/03/23 18:37:53 INFO session.SessionState: Created local directory: /tmp/920b853e-3aa6-4605-8ea5-1e3e67cbf574_resources
22/03/23 18:37:53 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/920b853e-3aa6-4605-8ea5-1e3e67cbf574
22/03/23 18:37:53 INFO session.SessionState: Created local directory: /tmp/hadoop/920b853e-3aa6-4605-8ea5-1e3e67cbf574
22/03/23 18:37:53 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/920b853e-3aa6-4605-8ea5-1e3e67cbf574/_tmp_space.db
22/03/23 18:37:53 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala> spark.conf.set("spark.sql.sources.default","csv")
vsailesh
commented
1 year ago Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/26 09:25:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x29647f75) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x29647f75
at org.apache.spark.storage.StorageUtils$.
I try to run spark-shell and get this error
I create cluster for testing using this command
clusterdock_run ./bin/start_cluster -n spark cdh --primary-node=node1 --secondary-nodes=node2