search

JanusGraph / janusgraph

JanusGraph: an open-source, distributed graph database
https://janusgraph.org
Other
5.32k stars 1.17k forks source link

JanusGraph/Bigtable integration failing with current Bigtable version #2060

Closed kolea2 closed 4 years ago

kolea2 commented 4 years ago

Please post all questions and issues on janusgraph-users before opening a GitHub issue. Your questions will reach a wider audience there, and if we confirm that there is a bug, then you can open a new issue.

Please include configurations and logs if available.

For confirmed bugs, please report:

When trying to load my .properties file (graph = JanusGraphFactory.open('conf/test.properties')), I am also seeing the error mentioned in the mailing list thread (below). My properties file is identical to the suggested changes to work with Bigtable here: https://docs.janusgraph.org/storage-backend/bigtable/

Stack Trace (if you have one)

java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.hbase.HBaseStoreManager
    at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
    at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
    at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:411)
    at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:50)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:79)
    at org.janusgraph.core.JanusGraphFactory$open.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at groovysh_evaluate.run(groovysh_evaluate:3)
    at groovysh_evaluate$run.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at groovysh_evaluate$run.call(Unknown Source)
    at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:81)
    at org.codehaus.groovy.tools.shell.Evaluator$evaluate.call(Unknown Source)
    at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:201)

... (full stack trace in thread)

Caused by: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
    ... 64 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent failure in storage backend
    at org.janusgraph.diskstorage.hbase.HBaseStoreManager.<init>(HBaseStoreManager.java:338)
    ... 69 more
Caused by: java.io.IOException: java.lang.NoSuchMethodException: com.google.cloud.bigtable.hbase1_x.BigtableConnection.<init>(org.apache.hadoop.conf.Configuration, java.util.concurrent.ExecutorService, org.apache.hadoop.hbase.security.User)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:222)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:115)
    at org.janusgraph.diskstorage.hbase.HBaseCompat1_0.createConnection(HBaseCompat1_0.java:43)
    at org.janusgraph.diskstorage.hbase.HBaseStoreManager.<init>(HBaseStoreManager.java:336)
    ... 69 more
Caused by: java.lang.NoSuchMethodException: com.google.cloud.bigtable.hbase1_x.BigtableConnection.<init>(org.apache.hadoop.conf.Configuration, java.util.concurrent.ExecutorService, org.apache.hadoop.hbase.security.User)
    at java.base/java.lang.Class.getConstructor0(Class.java:3350)
    at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2554)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:217)
    ... 72 more

This is happening because in the last part of the stacktrace, the Bigtable Hbase 1.x client does not have a constructor mapping to the parameters listed which are called reflectively. (java.lang.NoSuchMethodException: com.google.cloud.bigtable.hbase1_x.BigtableConnection.<init>(org.apache.hadoop.conf.Configuration, java.util.concurrent.ExecutorService, org.apache.hadoop.hbase.security.User))

To fix this, I'd like to propose upgrading the bigtable-hbase-1.x-shaded dependency to the bigtable-hbase-2.x-shaded version, which is compatible with the version of hbase and will not throw this error. I will create a PR for this unless there are any other ideas or objections. Thanks!

korjavin commented 4 years ago

I tried one more time with fresh start and looong pauses, and got this after query:

by org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, change order of serialization configuration if this is not desired.
70105 [gremlin-server-boss-1] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
70105 [gremlin-server-boss-1] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Channel started at port 8182.
130598 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - New session established for 0c244d5a-d2b3-4948-842b-70a80253c90e
140425 [gremlin-server-session-1] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Lock write succeeded but took too long: duration PT0.155S exceeded limit PT0.1S
140609 [gremlin-server-session-1] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Lock write succeeded but took too long: duration PT0.184S exceeded limit PT0.1S
140769 [gremlin-server-session-1] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Lock write succeeded but took too long: duration PT0.16S exceeded limit PT0.1S
140944 [gremlin-server-session-1] ERROR org.janusgraph.graphdb.database.StandardJanusGraph  - Could not commit transaction [1] due to exception
org.janusgraph.diskstorage.locking.TemporaryLockingException: Temporary locking failure
    at org.janusgraph.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:309)
    at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:103)
    at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:51)
    at org.janusgraph.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:255)
    at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:591)
    at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:712)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1425)
    at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:195)
    at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:111)
    at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:65)
    at org.janusgraph.example.GraphOfTheGodsFactory$load.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at Script4.run(Script4.groovy:1)
    at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:674)
    at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
    at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
    at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Lock write retry count exceeded
    at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:335)
    at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:118)
    at org.janusgraph.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:304)
    ... 24 more
140947 [gremlin-server-session-1] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Exception processing a script on request [RequestMessage{, requestId=4850867f-6f9d-42bb-abef-ed96758f232c, op='eval', processor='session', args={gremlin=GraphOfTheGodsFactory.load(graph), session=0c244d5a-d2b3-4948-842b-70a80253c90e, manageTransaction=false, userAgent=Gremlin Console/3.4.6, batchSize=64}}].
org.janusgraph.diskstorage.TemporaryBackendException: Lock write retry count exceeded
    at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:335)
    at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:118)
    at org.janusgraph.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:304)
    at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:103)
    at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:51)
    at org.janusgraph.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:255)
    at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:591)
    at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:712)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1425)
    at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:195)
    at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:111)
    at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:65)
    at org.janusgraph.example.GraphOfTheGodsFactory$load.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at Script4.run(Script4.groovy:1)
    at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:674)
    at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
    at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
    at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I hope it can help a bit.

korjavin commented 4 years ago

Any advice?