spmallette
commented
10 years ago @dalaro @okram any ideas what this is? i don't do much with hbase/faunus so the errors look sorta foreign to me.
Open varunkalluri opened 10 years ago
spmallette
commented
10 years ago @dalaro @okram any ideas what this is? i don't do much with hbase/faunus so the errors look sorta foreign to me.
dalaro
commented
10 years ago Based on this recent aureliusgraphs list discussion, I wonder if the problem was related to customized ZK root node (/hbase-unsecure): https://groups.google.com/d/msg/aureliusgraphs/t1FJQ_EByYg/EB2Z4a6UWdsJ. It seems like you're already setting the zookeeper.znode.parent key that resolved the issue in that thread, so maybe there's some other complication here. The connection is faint if it exists at all. There's not enough information in the trace to definitively say why the HBase client in Titan failed to find the HMaster (assuming the master was running). Can you turn up the logging level on the failing tasks to DEBUG? It would be interesting to see what they emit before dying with that exception.
I'm closing this ticket since I expect that you've moved on since July, but please do reopen if you want to pursue this further.
dalaro
commented
10 years ago Per your colleague Mirco Manucci's recent comment related to this issue on http://stackoverflow.com/questions/24886501/issue-while-ingesting-a-titan-graph-into-faunus directing me back to this issue, I'm reopening to reply and to try to gather this discussion into a single thread. I don't know if the SO question and this issue are crosspost of an identical problem, or if you guys are doing different work and encountering related problems, or something else. But Mirco directed me here when I asked for a full stacktrace in SO comments, so I assume moving the discussion here is OK.
Mirco or Varun, if you can turn up the logging level on the dying MR tasks to DEBUG to see why it's failing to connect to the HBase master, that would be really helpful. Right now, all I can see is that the task running Titan-HBase can't talk to the master, and I can't say why without more information.
varunkalluri
commented
10 years ago Hi LaRocque, Thanks for your response, Right now we are integrated titan-0.5.0 but gremlin shell is hanging with our hbase 0.94 version. I see that titan has 0.98 hbase jar files, I tried to copy libs from existing cluster still doesn't work .
Configuration file. storage.backend=hbase storage.hostname=xxxxxxx,xxxxxxxx,xxxxxxxx storage.port=2181 storage.short-cf-names=true storage.tablename=titan123 cache.db-cache = true cache.db-cache-clean-wait = 20 cache.db-cache-time = 180000 cache.db-cache-size = 0.5 storage.index.search.backend=elasticsearch storage.index.search.directory=/tmp/searchindex storage.index.search.client-only=false storage.index.search.local-mode=true
Tue Sep 23 11:16:15 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true Tue Sep 23 11:17:35 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true Tue Sep 23 11:18:55 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true Tue Sep 23 11:20:15 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true Tue Sep 23 11:21:35 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true Tue Sep 23 11:22:55 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true
11:22:55 WARN com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager - Unexpected exception during getDeployment()
java.lang.RuntimeException: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:346)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:384)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.
varunkalluri
commented
10 years ago We are using CDH4.5 hadoop version.
dalaro
commented
10 years ago @varunkalluri I'm a bit confused. The opening comment in this issue was about Faunus 0.4. Your later comment includes a call to TitanFactory, which isn't part of Faunus; it also mentions Titan 0.5, but the config file appears to be written for Titan 0.4 (the index keys have changed between 0.4 and 0.5).
Let's pursue Faunus 0.4 bugs here, but please file bugs related to 0.5 (whether Titan or Faunus) at https://github.com/thinkaurelius/titan/issues. There's also the https://groups.google.com/d/forum/aureliusgraphs list in case you prefer a discussion format.
Thanks!
Hi Okram, I have integrated faunus with hdfs and titan with hbase, But when we are trying to access data from titan into faunus it giving following error, Titan Version:titan-0.4.4 Faunus Version: faunus-0.4.4
Here is my property file: faunus.graph.input.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat faunus.graph.input.titan.storage.backend=hbase faunus.graph.input.titan.storage.hostname=xyz( I am specifying zookeeper hosts here) faunus.graph.input.titan.storage.port=2181 faunus.graph.input.titan.storage.tablename=titan
faunus.graph.output.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseOutputFormat faunus.graph.output.titan.storage.backend=hbase faunus.graph.output.titan.storage.hostname=xyz( I am specifying zookeeper hosts here) faunus.graph.output.titan.storage.port=2181 faunus.graph.output.titan.storage.tablename=titan faunus.graph.output.titan.storage.batch-loading=true faunus.output.location=output1 zookeeper.znode.parent=/hbase-unsecure
titan.graph.output.ids.block-size=100000
faunus.graph.output.titan.infer-schema=true
Execution: g = FaunusFactory.open('/opt/faunus/faunus-0.4.4/bin/titan-hbase-inputoutput.properties') ==>faunusgraph[titanhbaseinputformat->titanhbaseoutputformat] gremlin> g.v(2640092)
13:24:20 WARN mapreduce.FaunusCompiler: Using the distribution Faunus job jar: ../lib/faunus-0.4.4-job.jar 13:24:21 INFO mapreduce.FaunusCompiler: Compiled to 3 MapReduce job(s) 13:24:21 INFO mapreduce.FaunusCompiler: Executing job 1 out of 3: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VertexMap.Map, com.thinkaurelius.faunus.formats.titan.SchemaInferencerMapReduce.Map, com.thinkaurelius.faunus.formats.titan.SchemaInferencerMapReduce.Reduce] 13:24:21 INFO mapreduce.FaunusCompiler: Job data location: output1/job-0 13:24:21 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 13:24:22 INFO diskstorage.Backend: Initiated backend operations thread pool of size 4 13:24:22 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13:24:22 INFO zookeeper.ZooKeeper: Client environment:host.name=euca-192-168-216-107.eucalyptus.internal.devlab.dev 13:24:22 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_45 13:24:22 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 13:24:22 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_45/jre
13:24:24 INFO mapred.JobClient: Running job: job_201407181049_0009 13:24:25 INFO mapred.JobClient: map 0% reduce 0% 13:30:04 INFO mapred.JobClient: Task Id : attempt_201407181049_0009_m_000000_0, Status : FAILED com.thinkaurelius.titan.core.TitanException: Exception in Titan at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getAdminInterface(HBaseStoreManager.java:380) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:275) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:228) at com.thinkaurelius.titan.diskstorage.Backend.getStore(Backend.java:231) at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:243) at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1174) at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.(StandardTitanGraph.java:75) at com.thinkaurelius.faunus.formats.titan.FaunusTitanGraph.(FaunusTitanGraph.java:30) at com.thinkaurelius.faunus.formats.titan.FaunusTitanGraph.(FaunusTitanGraph.java:26) at com.thinkaurelius.faunus.formats.titan.hbase.FaunusTitanHBaseGraph.(FaunusTitanHBaseGraph.java:29) at com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat.setConf(TitanHBaseInputFormat.java:53) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Retried 14 times at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:139) at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getAdminInterface(HBaseStoreManager.java:378) ... 19 more 13:46:45 INFO mapred.JobClient: Job complete: job_201407181049_0009 13:46:45 INFO mapred.JobClient: Counters: 8 13:46:45 INFO mapred.JobClient: Job Counters 13:46:45 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1334254 13:46:45 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13:46:45 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13:46:45 INFO mapred.JobClient: Rack-local map tasks=3 13:46:45 INFO mapred.JobClient: Launched map tasks=4 13:46:45 INFO mapred.JobClient: Data-local map tasks=1 13:46:45 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 13:46:45 INFO mapred.JobClient: Failed map tasks=1 13:46:45 ERROR mapreduce.FaunusCompiler: Faunus job error -- remaining MapReduce jobs have been canceled
please let me know if I am missing any configutions(or) settings.
Thanks, varun