Titan + HBase hangs at Gremlin console

[cglewis]

Running on Ubuntu 12.04 LTS x86_64

Here's my maven version:

    user@host:~/titan-0.2.0$ mvn -v
    Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 13:51:28+0000)
    Maven home: /opt/apache-maven-3.0.5
    Java version: 1.6.0_32, vendor: Sun Microsystems Inc.
    Java home: /usr/java/jdk1.6.0_32/jre
    Default locale: en_US, platform encoding: UTF-8
    OS name: "linux", version: "3.2.0-27-virtual", arch: "amd64", family: "unix"

And running configuration for my remote HBase cluster (running CDH4.2 with Hbase 0.94.2-cdh4.2.0 and Titan 0.2.0):

    user@host:~/titan-0.2.0$ ./bin/gremlin.sh

             \,,,/
             (o o)
    -----oOOo-(_)-oOOo-----
    gremlin> conf = new BaseConfiguration();                                                                                                                                    
    ==>org.apache.commons.configuration.BaseConfiguration@1dbef987
    gremlin> conf.setProperty("storage.backend","hbase");                                                                                                                       
    ==>null
    gremlin> conf.setProperty("storage.hostname","x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x,x.x.x.x");
    ==>null
    gremlin> TitanGraph g = TitanFactory.open(conf);                                                                                                                            
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:host.name=host
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_32
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.6.0_32/jre
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.class.path=./bin/../lib/activation-1.1.jar:./bin/../lib/antlr-2.7.7.jar:./bin/../lib/antlr-3.2.jar:./bin/../lib/antlr-runtime-3.2.jar:./bin/../lib/asm-3.2.jar:./bin/../lib/asm-analysis-3.2.jar:./bin/../lib/asm-commons-3.2.jar:./bin/../lib/asm-tree-3.2.jar:./bin/../lib/asm-util-3.2.jar:./bin/../lib/astyanax-1.0.6.jar:./bin/../lib/avro-1.4.0-cassandra-1.jar:./bin/../lib/avro-ipc-1.5.3.jar:./bin/../lib/blueprints-core-2.2.0.jar:./bin/../lib/cassandra-all-1.1.3.jar:./bin/../lib/cassandra-thrift-1.1.3.jar:./bin/../lib/colt-1.2.0.jar:./bin/../lib/commons-beanutils-1.7.0.jar:./bin/../lib/commons-beanutils-core-1.8.0.jar:./bin/../lib/commons-cli-1.2.jar:./bin/../lib/commons-codec-1.4.jar:./bin/../lib/commons-collections-3.2.1.jar:./bin/../lib/commons-configuration-1.6.jar:./bin/../lib/commons-digester-1.8.jar:./bin/../lib/commons-el-1.0.jar:./bin/../lib/commons-httpclient-3.1.jar:./bin/../lib/commons-io-2.0.1.jar:./bin/../lib/commons-lang-2.4.jar:./bin/../lib/commons-logging-1.1.1.jar:./bin/../lib/commons-math-2.1.jar:./bin/../lib/commons-net-1.4.1.jar:./bin/../lib/commons-pool-1.5.5.jar:./bin/../lib/compress-lzf-0.8.4.jar:./bin/../lib/concurrent-1.3.4.jar:./bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:./bin/../lib/core-3.1.1.jar:./bin/../lib/gmbal-3.2.0-b003.jar:./bin/../lib/gremlin-groovy-2.2.0.jar:./bin/../lib/gremlin-java-2.2.0.jar:./bin/../lib/grizzly-core-2.2.18.jar:./bin/../lib/grizzly-framework-2.2.18.jar:./bin/../lib/grizzly-http-2.2.16.jar:./bin/../lib/grizzly-http-server-2.2.16.jar:./bin/../lib/grizzly-http-servlet-2.2.18.jar:./bin/../lib/grizzly-portunif-2.2.18.jar:./bin/../lib/grizzly-rcm-2.2.18.jar:./bin/../lib/groovy-1.8.8.jar:./bin/../lib/guava-12.0.jar:./bin/../lib/hadoop-core-1.0.3.jar:./bin/../lib/hbase-0.94.1.jar:./bin/../lib/high-scale-lib-1.1.2.jar:./bin/../lib/hsqldb-1.8.0.10.jar:./bin/../lib/httpclient-4.1.2.jar:./bin/../lib/httpcore-4.1.3.jar:./bin/../lib/jackson-core-asl-1.8.5.jar:./bin/../lib/jackson-jaxrs-1.8.8.jar:./bin/../lib/jackson-mapper-asl-1.8.5.jar:./bin/../lib/jackson-xc-1.8.8.jar:./bin/../lib/jamm-0.2.5.jar:./bin/../lib/jamon-runtime-2.3.1.jar:./bin/../lib/jansi-1.5.jar:./bin/../lib/jasper-compiler-5.5.23.jar:./bin/../lib/jasper-runtime-5.5.23.jar:./bin/../lib/javassist-3.15.0-GA.jar:./bin/../lib/javax.servlet-api-3.0.1.jar:./bin/../lib/jaxb-api-2.1.jar:./bin/../lib/jaxb-impl-2.2.3-1.jar:./bin/../lib/je-5.0.58.jar:./bin/../lib/jersey-client-1.14.jar:./bin/../lib/jersey-core-1.8.jar:./bin/../lib/jersey-grizzly2-1.14.jar:./bin/../lib/jersey-json-1.11.jar:./bin/../lib/jersey-server-1.14.jar:./bin/../lib/jersey-servlet-1.14.jar:./bin/../lib/jets3t-0.7.1.jar:./bin/../lib/jettison-1.3.jar:./bin/../lib/jetty-6.1.26.jar:./bin/../lib/jetty-util-6.1.26.jar:./bin/../lib/jline-0.9.94.jar:./bin/../lib/joda-time-2.0.jar:./bin/../lib/json-simple-1.1.jar:./bin/../lib/jsp-2.1-6.1.14.jar:./bin/../lib/jsp-api-2.1-6.1.14.jar:./bin/../lib/jsr305-1.3.9.jar:./bin/../lib/kfs-0.3.jar:./bin/../lib/kryo-1.04.jar:./bin/../lib/libthrift-0.8.0.jar:./bin/../lib/log4j-1.2.16.jar:./bin/../lib/management-api-3.0.0-b012.jar:./bin/../lib/metrics-core-2.1.2.jar:./bin/../lib/minlog-1.2.jar:./bin/../lib/msgpack-0.6.5.jar:./bin/../lib/netty-3.2.4.Final.jar:./bin/../lib/org.apache.servicemix.bundles.commons-csv-1.0-r706900_3.jar:./bin/../lib/oro-2.0.8.jar:./bin/../lib/pfl-asm-1.0.0-b001.jar:./bin/../lib/pfl-basic-1.0.0-b001.jar:./bin/../lib/pfl-basic-tools-1.0.0-b001.jar:./bin/../lib/pfl-tf-1.0.0-b001.jar:./bin/../lib/pfl-tf-tools-1.0.0-b001.jar:./bin/../lib/pipes-2.2.0.jar:./bin/../lib/protobuf-java-2.4.0a.jar:./bin/../lib/reflectasm-1.01.jar:./bin/../lib/rexster-core-2.2.0.jar:./bin/../lib/rexster-protocol-2.2.0.jar:./bin/../lib/rexster-server-2.2.0.jar:./bin/../lib/servlet-api-2.5-6.1.14.jar:./bin/../lib/slf4j-api-1.6.1.jar:./bin/../lib/slf4j-log4j12-1.6.1.jar:./bin/../lib/snakeyaml-1.6.jar:./bin/../lib/snappy-java-1.0.4.1.jar:./bin/../lib/snaptree-0.1.jar:./bin/../lib/stax-api-1.0.1.jar:./bin/../lib/stringtemplate-3.2.jar:./bin/../lib/titan-0.2.0.jar:./bin/../lib/uuid-3.2.0.jar:./bin/../lib/velocity-1.7.jar:./bin/../lib/xmlenc-0.52.jar:./bin/../lib/zookeeper-3.4.3.jar:.:
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/jdk1.6.0_32/jre/lib/amd64/server:/usr/java/jdk1.6.0_32/jre/lib/amd64:/usr/java/jdk1.6.0_32/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-27-virtual
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:user.name=user
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/user
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/user/titan-0.2.0
    13/02/27 19:31:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=x.x.x.x:2181 sessionTimeout=180000 watcher=hconnection
    13/02/27 19:31:40 INFO zookeeper.ClientCnxn: Opening socket connection to server /x.x.x.x:2181
    13/02/27 19:31:40 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 14114@host
    13/02/27 19:31:40 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
    13/02/27 19:31:40 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
    13/02/27 19:31:40 INFO zookeeper.ClientCnxn: Socket connection established to hbase-master/x.x.x.x:2181, initiating session
    13/02/27 19:31:40 INFO zookeeper.ClientCnxn: Session establishment complete on server hbase-master/x.x.x.x:2181, sessionid = 0x13d1d1a0766002f, negotiated timeout = 120000
    13/02/27 19:31:41 INFO client.HBaseAdmin: Started disable of titan
    13/02/27 19:31:43 INFO client.HBaseAdmin: Disabled titan
    13/02/27 19:31:45 INFO client.HBaseAdmin: Started enable of titan

Unfortunately at this point the console hangs (I can still control-c out of gremlin).

Any suggestions on what I might be missing or doing wrong? Prior to hanging, it does create the empty Titan table in HBase.

See also: #64

[cglewis]

Incidentally solved this. Was an NTP issue.

[cglewis]

As a side note, I also experienced this issue when running gremlin inside of tmux.

[SteamShon]

I am having exactly same problem here. can any one give me what is problem ans fix?

[cglewis]

Are you using CDH? I've found that with CDH you have to run the configuration about 6 times and then it works after that. If you are watching your HBase cluster, you'll notice the table being made, and each time you run the configuration another column family gets added. Once all of the column families are added, it works as expected. Or at least that has been my experience so far. I haven't been able to replicate this issue with HDP, and if you recompile Titan with CDH's hadoop-core, etc., this problem also goes away.

[simplyzaki]

@cglewis Was facing the same issue, but your stated trick ( 6 times config re-run ) got it working.

[AlexanderDaw]

Also seeing this issue. CDH4 , CentOS6.2.

[mphuff]

@cglewis, @AlexanderDaw, and @khanio -- any other idea as to what may be causing this? We are running CDH4 on Ubuntu 12.04 and I'm seeing exactly the same output as mentioned above. I've tried re-deploying the configuration through the Cloudera manager and that hasn't improved anything either. I have also tried explicitly including the hadoop-core library specific to my version of CDH4 and everything still hangs on the same line.

13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:host.name=hnode2.tuk2.intelius.com 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_27 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.class.path=xdm-1.11-SNAPSHOT-jar-with-dependencies.jar 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:java.compiler= 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-48-generic 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:user.name=mhuff 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/mhuff 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/mhuff 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hnode2.tuk2.intelius.com:2181 sessionTimeout=180000 watcher=hconnection 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: Opening socket connection to server hnode2.tuk2.intelius.com/10.199.2.242:2181 13/07/03 07:12:47 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 32584@hnode2.tuk2.intelius.com 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: Socket connection established to hnode2.tuk2.intelius.com/10.199.2.242:2181, initiating session 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: Session establishment complete on server hnode2.tuk2.intelius.com/10.199.2.242:2181, sessionid = 0x13f594bff73a85c, negotiated timeout = 60000 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hnode2.tuk2.intelius.com:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@620a3d3b 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: Opening socket connection to server hnode2.tuk2.intelius.com/10.199.2.242:2181 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: Socket connection established to hnode2.tuk2.intelius.com/10.199.2.242:2181, initiating session 13/07/03 07:12:47 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 32584@hnode2.tuk2.intelius.com 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: Session establishment complete on server hnode2.tuk2.intelius.com/10.199.2.242:2181, sessionid = 0x13f594bff73a85d, negotiated timeout = 60000 13/07/03 07:12:47 INFO zookeeper.ZooKeeper: Session: 0x13f594bff73a85d closed 13/07/03 07:12:47 INFO zookeeper.ClientCnxn: EventThread shut down 13/07/03 07:12:48 INFO client.HBaseAdmin: Started disable of xdm 13/07/03 07:12:50 INFO client.HBaseAdmin: Disabled xdm 13/07/03 07:12:51 INFO client.HBaseAdmin: Started enable of xdm

[simplyzaki]

@mphuff What version of Titan are you using ?

[mphuff]

I'm using the latest version of Titan, 0.3.1. I've tried using titan-hbase as well as titan-all with no difference as well.

• Micah

On Jul 4, 2013, at 3:48 AM, Zakiullah Khan Mohammed notifications@github.com wrote:

@mphuff What version of Titan are you using ?

— Reply to this email directly or view it on GitHub.

[simplyzaki]

@mphuff And is CDH4 installed in cluster mode or pesudo (standalone) mode ?

[mphuff]

It is in clustered mode.

• Micah

On Jul 4, 2013, at 8:13 AM, Zakiullah Khan Mohammed notifications@github.com wrote:

@mphuff And is CDH4 installed in cluster mode or pesudo (standalone) mode ?

— Reply to this email directly or view it on GitHub.

[simplyzaki]

And when referencing the HBase server setting in Titan config, are you pointing to CDH4 client node or HBASE master node ?

[mphuff]

@khanio -- I'm pretty sure I'm pointing to the HBase master node. Both CDH4 and HBase are running on the same master machine and the region servers for hbase are running on 2 other servers. This is a cluster we're using as a test setup and we're trying to understand what may be wrong -- either with Titan or HBase itself.

[mphuff]

Just as a follow-up, I've been successfully able to create a table just through the hbase shell, insert records, add a number of column families, and everything worked as expected. When I run the following SIMPLE java code though, it stalls at 'Started enable of xdm':

Configuration conf = new BaseConfiguration();
conf.setProperty("storage.backend", "hbase");
conf.setProperty("storage.tablename", "xdm");
conf.setProperty("storage.hostname", "hnode2.tuk2.intelius.com");
TitanGraph graph = TitanFactory.open(conf);
graph.shutdown();

[okram]

Does it "stall" or does it go into what looks as an infinite loop. If the latter, realize that Titan will enable/disable/enable/disable/etc. for a good 45 seconds before completing and table creation is complete.

On Jul 5, 2013, at 10:55 AM, Micah Huff notifications@github.com wrote:

Just as a follow-up, I've been successfully able to create a table just through the hbase shell, insert records, add a number of column families, and everything worked as expected. When I run the following SIMPLE java code though, it stalls at 'Started enable of xdm':

Configuration conf = new BaseConfiguration(); conf.setProperty("storage.backend", "hbase"); conf.setProperty("storage.tablename", "xdm"); conf.setProperty("storage.hostname", "hnode2.tuk2.intelius.com"); TitanGraph graph = TitanFactory.open(conf); graph.shutdown(); — Reply to this email directly or view it on GitHub.

[mphuff]

@okram -- It appears to stall. I only ever see one iteration of 'start disable', 'disabled', 'start enable' and then nothing else. I just recently let it sit there at the 'start enable' line for as long as it would and I wound up with this call stack as the resulting error:

Exception in thread "main" com.thinkaurelius.titan.core.TitanException: Could not initialize backend
    at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:210)
    at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:441)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:67)
    at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
    at com.inome.tinkerpopgraph.InitializeGraph.main(InitializeGraph.java:20)
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:226)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:170)
    at com.thinkaurelius.titan.diskstorage.Backend.getStore(Backend.java:162)
    at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:173)
    ... 4 more
Caused by: java.io.IOException: Unable to enable table xdm
    at org.apache.hadoop.hbase.client.HBaseAdmin.enableTable(HBaseAdmin.java:675)
    at org.apache.hadoop.hbase.client.HBaseAdmin.enableTable(HBaseAdmin.java:633)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:219)
    ... 7 more

[mphuff]

@okram, @cglewis, @AlexanderDaw, and @khanio --

Update as to where I'm at. It appears that this is a compatibility problem between CDH4.3.0 and Titan 0.3.1. I'm still working out the problem but I'm now explicitly including all of the CDH4.3.0 hadoop binaries (hadoop-core and hadoop-common) as well as the hbase 0.94.6-cdh4.3.0 binaries.

By doing that, I don't get all the way to the point in time where the table is going to be created but here is the updated output that I'm now working through:

mhuff@hnode2:~$ java -cp xdm-1.11-SNAPSHOT-jar-with-dependencies.jar com.inome.tinkerpopgraph.InitializeGraph
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:host.name=hnode2.tuk2.intelius.com
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_27
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.class.path=xdm-1.11-SNAPSHOT-jar-with-dependencies.jar
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-48-generic
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:user.name=mhuff
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/mhuff
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/mhuff
13/07/08 17:14:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hnode2.tuk2.intelius.com:2181 sessionTimeout=180000 watcher=hconnection
13/07/08 17:14:51 INFO zookeeper.ClientCnxn: Opening socket connection to server hnode2.tuk2.intelius.com/10.199.2.242:2181
13/07/08 17:14:51 INFO zookeeper.ClientCnxn: Socket connection established to hnode2.tuk2.intelius.com/10.199.2.242:2181, initiating session
13/07/08 17:14:51 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 5869@hnode2.tuk2.intelius.com
13/07/08 17:14:51 INFO zookeeper.ClientCnxn: Session establishment complete on server hnode2.tuk2.intelius.com/10.199.2.242:2181, sessionid = 0x13f594bff73e678, negotiated timeout = 60000
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:437)
    at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
    at sun.proxy.$Proxy5.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:711)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:126)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getAdminInterface(HBaseStoreManager.java:327)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:198)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:170)
    at com.thinkaurelius.titan.diskstorage.Backend.getStore(Backend.java:162)
    at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:173)
    at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:441)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:67)
    at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
    at com.inome.tinkerpopgraph.InitializeGraph.main(InitializeGraph.java:22)

I'm hoping that I should have more information tomorrow but I thought I would put this out there in case anybody has seen this error previously.

Thanks!

[cglewis]

@mphuff I haven't tried with the latest yet so there be some differences to making it work. Are you creating a new HBase table before letting Titan do it, or is Titan creating it for you? I've had issues with pre-existing tables and the CDH hack, so would be curious what you're doing.

[mphuff]

@cglewis - I am letting Titan do the creation of the table for me. I want to have no manual Hbase interaction and let Titan take care of everything. I posted the code I'm using just a few posts ago. I never make it to the shutdown line.

[erickt]

Reconfirmed this is still happening on CDH 4.3.1 and Titan 0.3.2. 6 times gets it working.

EDIT: Had the wrong CDH version.

[mbroecheler]

@erickt: Does that mean it works with a more recent version or that the version number was originally wrong? Does this ticket need to be reopened?

[erickt]

@mbroecheler: I didn't notice that this issue was closed. Yes, this ticket should be reopened.

[dkuppitz]

Just stumbled across this issue. Maybe the following piece of information will help somebody.

I started the Gremlin REPL on the HBase master, tried g = TitanFactory.open("bin/hbase.local") and the console was hanging as described a few times here. However, everything worked just fine after I've changed the hostname in hbase.local. These configurations did not work (console hanging):

storage.backend=hbase
storage.hostname=<public ip of master node>

storage.backend=hbase
storage.hostname=<private ip of master node>

storage.backend=hbase
storage.hostname=localhost

storage.backend=hbase

This configuration did work:

storage.backend=hbase
storage.hostname=127.0.0.1

[kkochis]

I've also come across this issue and have tried several scenarios which do not work. Apache HBase (non-Cloudera) as well as MapR's HBase also have this same issue described. I am not using a local HBase install so dkuppitz's workaround does not work for me. I tried Titan versions 0.3.2 and 0.3.0 without success.

[spmallette]

last mentions of these problems go back to titan 0.3.x - Is this still an issue as of Titan 1.0? @graben1437 you do a fair bit with hbase - can you confirm this as something we could close?

[graben1437]

@spmallette I will take a look and let you know what I see.