search

Huawei-Spark / Spark-SQL-on-HBase

Native, optimized access to HBase Data through Spark SQL/Dataframe Interfaces
Apache License 2.0
321 stars 164 forks source link

Issue while running Spark-sql on Hbase #20

Closed rkiyer999 closed 9 years ago

rkiyer999 commented 9 years ago

I am working on Spark-sql on HBase : I am working on Horton works 2.3 VM which supports spark 1.3.1 so I externally downloaded spark 1.4.0.

Hbase version is :Version 1.1.0.2.3.0.0-2130

I installed Spark SQL on Hbase based on instruction specified in : https://github.com/Huawei-Spark/Spark-SQL-on-HBase

--More--[root@sandbox bin]# ./hbase-sql 15/10/07 06:15:47 INFO spark.SparkContext: Running Spark version 1.4.0 15/10/07 06:15:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/10/07 06:15:48 INFO spark.SecurityManager: Changing view acls to: root 15/10/07 06:15:48 INFO spark.SecurityManager: Changing modify acls to: root 15/10/07 06:15:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 15/10/07 06:15:49 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/10/07 06:15:49 INFO Remoting: Starting remoting 15/10/07 06:15:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.0.2.15:49136] 15/10/07 06:15:49 INFO util.Utils: Successfully started service 'sparkDriver' on port 49136. 15/10/07 06:15:50 INFO spark.SparkEnv: Registering MapOutputTracker 15/10/07 06:15:50 INFO spark.SparkEnv: Registering BlockManagerMaster 15/10/07 06:15:50 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-3a74e54e-caba-40c5-90a9-918be1e9ad99/blockmgr-2e0275db-ea0e-4c36-a586-ff866f575271 15/10/07 06:15:50 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB 15/10/07 06:15:50 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-3a74e54e-caba-40c5-90a9-918be1e9ad99/httpd-b9ae55de-0e91-4488-9127-20ec61d563eb 15/10/07 06:15:50 INFO spark.HttpServer: Starting HTTP Server 15/10/07 06:15:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/10/07 06:15:50 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:36460 15/10/07 06:15:50 INFO util.Utils: Successfully started service 'HTTP file server' on port 36460. 15/10/07 06:15:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/10/07 06:15:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/10/07 06:15:50 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/10/07 06:15:50 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/10/07 06:15:50 INFO ui.SparkUI: Started SparkUI at http://10.0.2.15:4040 15/10/07 06:15:50 INFO spark.SparkContext: Added JAR file:/home/rk/spark-hbase/spark-hbase/target/spark-sql-on-hbase-1.0.0.jar at http://10.0.2.15:36460/jars/spark-sql-on-hbase-1.0.0.jar with timestamp 1444198550958 15/10/07 06:15:51 INFO executor.Executor: Starting executor ID driver on host localhost 15/10/07 06:15:51 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48672. 15/10/07 06:15:51 INFO netty.NettyBlockTransferService: Server created on 48672 15/10/07 06:15:51 INFO storage.BlockManagerMaster: Trying to register BlockManager 15/10/07 06:15:51 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:48672 with 265.4 MB RAM, BlockManagerId(driver, localhost, 48672) 15/10/07 06:15:51 INFO storage.BlockManagerMaster: Registered BlockManager Welcome to hbaseql CLI astro> select * from emp; 15/10/07 06:16:12 INFO hbase.HBaseSQLCliDriver: Processing select * from emp 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:host.name=sandbox.hortonworks.com 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_79 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/rk/spark/spark-1.4.0-bin-hadoop2.4/conf/:/home/rk/spark/spark-1.4.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-hadoop2.4.0.jar:/home/rk/spark/spark-1.4.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/home/rk/spark/spark-1.4.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/home/rk/spark/spark-1.4.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/usr/hdp/2.3.0.0-2130/hadoop/conf/ 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:java.compiler= 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-504.16.2.el6.x86_64 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:user.name=root 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:user.home=/root 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/rk/spark-hbase/spark-hbase/bin 15/10/07 06:16:12 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x977faf, quorum=localhost:2181, baseZNode=/hbase 15/10/07 06:16:13 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x977faf connecting to ZooKeeper ensemble=localhost:2181 15/10/07 06:16:13 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 15/10/07 06:16:13 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 15/10/07 06:16:13 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15040d0f571001a, negotiated timeout = 40000 15/10/07 06:16:13 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null 15/10/07 06:16:14 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x977faf, quorum=localhost:2181, baseZNode=/hbase 15/10/07 06:16:14 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x977faf connecting to ZooKeeper ensemble=localhost:2181 15/10/07 06:16:14 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 15/10/07 06:16:14 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 15/10/07 06:16:14 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15040d0f571001b, negotiated timeout = 40000

My system doesn't respond

yzhou2001 commented 9 years ago

I suspect that HBase 1.1 can work with this product. HBase 1.0 is probably ok. Anyways, please make sure your HBase and Spark can work separately before try this product.