Closed kolea2 closed 4 years ago
mbrukman
commented
4 years ago @porunov — should this fix be targeted at master or the 0.5 branch?
I believe this bug is also present in the 0.4.x series (we've seen some issues raised on the mailing list); we may want to backport it to the 0.4 branch as well, what do you think?
porunov
commented
4 years ago @mbrukman We stopped 0.4 support. Of course we can merge fixes in older branches (i.e. 0.4), but we most likely won't make new releases from those branches unless there is a demand for those older versions.
Currently we should target this fix to 0.5 branch because we are currently supporting 0.5 and master (0.6) branches.
kolea2
commented
4 years ago Thanks @mbrukman, @porunov - in working on this and switching from the 1x to 2x version, users would have to update their configs (https://github.com/JanusGraph/janusgraph/blob/master/docs/storage-backend/bigtable.md) to storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection. Wanted to get your thoughts as this feels like a breaking change here.
korjavin
commented
4 years ago Changing to com.google.cloud.bigtable.hbase2_x.BigtableConnection (v0.4) I've got:
java.lang.RuntimeException: GraphFactory could not instantiate this Graph
implementation [class org.janusgraph.core.JanusGraphFactory]
at
org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at
org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at
org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at
org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at
org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at
org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at
org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at
org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 13 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate
implementation: org.janusgraph.diskstorage.hbase.HBaseStoreManager
at
org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at
org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:411)
at
org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:50)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 24 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent
failure in storage backend
at
org.janusgraph.diskstorage.hbase.HBaseStoreManager.<init>(HBaseStoreManager.java:337)
... 29 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.google.cloud.bigtable.hbase2_x.BigtableConnection
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:213)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:115)
at
org.janusgraph.diskstorage.hbase.HBaseCompat1_0.createConnection(HBaseCompat1_0.java:43)
at
org.janusgraph.diskstorage.hbase.HBaseStoreManager.<init>(HBaseStoreManager.java:335)
... 29 more
Caused by: java.lang.ClassNotFoundException:
com.google.cloud.bigtable.hbase2_x.BigtableConnection
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:211)
... 32 moreOn Fri, Mar 27, 2020 at 4:52 PM kolea2 notifications@github.com wrote:
Thanks @mbrukman https://github.com/mbrukman, @porunov https://github.com/porunov - in working on this and switching from the 1x to 2x version, users would have to update their configs ( https://github.com/JanusGraph/janusgraph/blob/master/docs/storage-backend/bigtable.md) to storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection. Wanted to get your thoughts as this feels like a breaking change here.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-605012583, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEKHNMG25AHVLYUFJYK3DRJSVTTANCNFSM4LUI25EQ .
mbrukman
commented
4 years ago @kolea2 – I think that's fine; I don't think any of the 0.4.x versions work with Bigtable, so folks upgrading to the 0.5.x series would be upgrading from 0.3.x or earlier, so it would be a conscious decision on their part, and we'll have to mention this in the release notes that they will have to update this parameter as well. Please update the docs in your PR as well.
@porunov, what do you think?
@korjavin – just changing that parameter on its own is not sufficient; @kolea2 is working on a change to the pom.xml files to add different Hadoop1 vs. Hadoop2 profiles which, coupled with the change mentioned above, will enable for you to use JanusGraph with Bigtable. Please stay tuned for the upcoming PR with the fix.
kolea2
commented
4 years ago @mbrukman - after testing with the hbase1.version (1.4.10): https://github.com/JanusGraph/janusgraph/blob/master/pom.xml#L84, and the current 1x bigtable hbase shaded jar, I'm running into new errors:
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Admin.addColumn(Lorg/apache/hadoop/hbase/TableName;Lorg/apache/hadoop/hbase/client/ColumnFamilyDescriptor;)V
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.addColumn(HBaseAdmin1_0.java:159)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:844)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:487)
at org.janusgraph.diskstorage.keycolumnvalue.KeyColumnValueStoreManager.openDatabase(KeyColumnValueStoreManager.java:43)
at org.janusgraph.diskstorage.Backend$5.apply(Backend.java:611)which looks like more work would be required in this codebase besides creating the maven profile and updating dependencies. This is why I'm unsure if we should work on that rather than simply switching the version to bigtable-hbase-2x-shaded. Let me know what you think or if my understanding is incorrect (this is my first time digging through this code :) )
porunov
commented
4 years ago @mbrukman @kolea2 yes, I think fixing it in 0.5.x version and mentioning it in the documentation is enough. We don't need to fix it in 0.4.x version
haroon-qureshi
commented
4 years ago Hello, I raised this issue on the group. We are working with v0.4.1 and trying to make it work with BigTable. According to the BigTable documentation (https://cloud.google.com/bigtable/docs/hbase-bigtable), “The client is compatible with versions 1.0.x, 1.1.x, 1.2.x, and 1.3.x of the HBase API”. I’m not sure if this was taken into consideration but just wanted to share this with you.
mbrukman
commented
4 years ago @haroon-qureshi — you're right, we should probably update the docs to specify that v0.4.1. does not work with Bigtable.
FWIW, PR #2064 fixes this issue on the v0.5 branch which means it will be in a future 0.5.x release. Would you be able to use that version in place of the 0.4.1 you're working with now?
haroon-qureshi
commented
4 years ago We can. Do you know when the fix will be released and which version it will be in?
Also, I think it affects the whole v0.4 branch, not just 0.4.1.
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Misha Brukman notifications@github.com Sent: Monday, March 30, 2020 10:19:15 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
@haroon-qureshihttps://github.com/haroon-qureshi — you're right, we should probably update the docs to specify that v0.4.1. does not work with Bigtable.
FWIW, PR #2064https://github.com/JanusGraph/janusgraph/pull/2064 fixes this issue on the v0.5 branch which means it will be in a future 0.5.x release. Would you be able to use that version in place of the 0.4.1 you're working with now?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-606063798, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVDLBY5BO5ANH2OMAFTRKCZ7HANCNFSM4LUI25EQ.
porunov
commented
4 years ago @haroon-qureshi the fix is targeting 0.5.1 release. Most likely it will be released this week (depends on the TSC voting process).
v0.4 is EOL. We don't plan to support that branch anymore. You should be able to upgrade to 0.5.x without any problems from 0.4.x unless you are running ElasticSearch version less than 6 (i.e. 5. In such case update and reindexing is required). You can see breaking changes here https://docs.janusgraph.org/changelog/#upgrade-instructions_1
porunov
commented
4 years ago Fixed in #2064
mbrukman
commented
4 years ago quoting @porunov on the PR #2064:
0.5.1release artefacts have been updated to include this PR. The release is uploaded into Sonatype staging repository and the tag and GitHub release jars have been updated as well (https://github.com/JanusGraph/janusgraph/releases/tag/v0.5.1).
@haroon-qureshi — could you please try out the pre-release v0.5.1 by using the artifacts from the above link and see if it resolves your problem?
korjavin
commented
4 years ago What about docker image?
mbrukman
commented
4 years ago @korjavin — I think the Docker image will only be updated after the release is approved, which hasn't happened yet, we're still in the testing/approval stage as you can see in https://groups.google.com/d/msg/janusgraph-dev/qQRc8fT5aPw/LbEJHvibBAAJ
mbrukman
commented
4 years ago @korjavin — if you are able to test the pre-release without the Docker image, please do, and let us know if it works for you. Thanks!
haroon-qureshi
commented
4 years ago Yes I can work on this tomorrow and get back to you.
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Misha Brukman notifications@github.com Sent: Tuesday, March 31, 2020 2:45:15 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
@korjavinhttps://github.com/korjavin — if you are able to test the pre-release without the Docker image, please do, and let us know if it works for you. Thanks!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-606835542, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVCZYEVK2V5EQYP7WO3RKJB4XANCNFSM4LUI25EQ.
korjavin
commented
4 years ago I use k8s, so it's hard for me to test without docker image. I'm trying to build it myself following readme, but not success-ed yet.
haroon-qureshi
commented
4 years ago I use k8s as well.
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Korjavin Ivan notifications@github.com Sent: Tuesday, March 31, 2020 2:46:47 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
I use k8s, so it's hard for me to test without docker image. I'm trying to build it myself following readme, but not success-ed yet.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-606836348, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVDE577T6MSV3IYYNOTRKJCCPANCNFSM4LUI25EQ.
mbrukman
commented
4 years ago @korjavin, @haroon-qureshi — you should be able to build the Docker images with 0.5.1 via:
git clone https://github.com/JanusGraph/janusgraph-dockercd janusgraph-docker./update.sh which will update the 0.5/Dockerfile versions with the latest 0.5.x version which is 0.5.10.5/Dockerfile — https://github.com/JanusGraph/janusgraph-docker/blob/master/build-images.sh has the actual build process with args, but you only need to build a single image so you can modify versions in the script or run the command(s) yourself directly.Hope this helps! We'll try to make this easier to test in the future.
haroon-qureshi
commented
4 years ago Thanks
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Misha Brukman notifications@github.com Sent: Tuesday, March 31, 2020 2:56:27 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
@korjavinhttps://github.com/korjavin, @haroon-qureshihttps://github.com/haroon-qureshi — you should be able to build the Docker images with 0.5.1 via:
Hope this helps! We'll try to make this easier to test in the future.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-606841519, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVE4E6DTWHQA6USS4NLRKJDGXANCNFSM4LUI25EQ.
korjavin
commented
4 years ago Wow, I've already built docker, trying to deploy it now.
mbrukman
commented
4 years ago We already have good instructions in https://github.com/JanusGraph/janusgraph-docker/blob/master/BUILDING.md, so you can do this even simpler than what I suggested earlier:
$ git clone https://github.com/JanusGraph/janusgraph-docker
$ cd janusgraph-docker
$ ./update.sh 0.5.1
$ ./build-images.sh 0.5And that's it!
korjavin
commented
4 years ago
storage.backend: hbase
storage.directory: null
storage.hbase.ext.google.bigtable.instance.id: {realid}
storage.hbase.ext.google.bigtable.project.id: {realproject}
storage.hbase.ext.hbase.client.connection.impl: com.google.cloud.bigtable.hbase2_x.BigtableConnection
index.search.backend: elasticsearch
index.search.directory: null
index.search.hostname: 192.168.1.69
cache.db-cache: true
cache.db-cache-clean-wait: 20
cache.db-cache-time: 180000
cache.db-cache-size: 0.5
ids.authority.wait-time: 0gremlin-server.yaml
host: 0.0.0.0 port: 8182 scriptEvaluationTimeout: 30000 channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer graphs: { graph: /etc/opt/janusgraph/janusgraph.properties } scriptEngines: { gremlin-groovy: { plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {}, org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {}, org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {}, org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]}, org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/empty-sample.groovy]}}}} serializers: ....
After start it looks good:2718 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com 2962 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtableadmin.googleapis.com 46138 [main] INFO org.janusgraph.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 13 OK out of 13 attempts in PT0.068S 54622 [main] INFO org.janusgraph.graphdb.idmanagement.UniqueInstanceIdRetriever - Generated unique-instance-id=ac11000330-9c77409ef4e31 54655 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection 54655 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting google.bigtable.project.id=... 54655 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting google.bigtable.instance.id=... 54657 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Opening session for projectId phonic-heaven-126923, instanceId live3pm, on data host bigtable.googleapis.com, admin host bigtableadmin.googleapis.com. 54657 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Bigtable options: BigtableOptions{dataHost=bigtable.googleapis.com, adminHost=bigtableadmin.googleapis.com, port=443, projectId=phonic-heaven-126923, instanceId=..., appProfileId=, userAgent=hbase-2.1.5, credentialType=DefaultCredentials, dataChannelCount=32, retryOptions=RetryOptions{retriesEnabled=true, allowRetriesWithoutTimestamp=false, statusToRetryOn=[ABORTED, UNAUTHENTICATED, DEADLINE_EXCEEDED, UNAVAILABLE], initialBackoffMillis=5, maxElapsedBackoffMillis=60000, backoffMultiplier=1.5, streamingBufferSize=60, readPartialRowTimeoutMillis=60000, maxScanTimeoutRetries=3}, bulkOptions=BulkOptions{asyncMutatorCount=2, useBulkApi=true, bulkMaxKeyCount=125, bulkMaxRequestSize=1048576, autoflushMs=0, maxInflightRpcs=320, maxMemory=47762636, enableBulkMutationThrottling=false, bulkMutationRpcTargetMs=100}, callOptionsConfig=CallOptionsConfig{useTimeout=false, shortRpcTimeoutMs=60000, longRpcTimeoutMs=600000, mutateRpcTimeoutMs=600000, readStreamRpcTimeoutMs=600000}, usePlaintextNegotiation=false, useCachedDataPool=false, useBatch=false, useGCJClient=false}
Then I connect with gremlingremlin> :remote connect tinkerpop.server conf/remote.yaml session ==>Configured localhost/127.0.0.1:8182-[7cba99a7-6238-42a3-871b-c14a2131ae6c] gremlin> :remote console ==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182]-[7cba99a7-6238-42a3-871b-c14a2131ae6c] - type ':remote console' to return to local mode gremlin> GraphOfTheGodsFactory.load(graph)
And logs:322524 [gremlin-server-worker-1] INFO org.apache.tinkerpop.gremlin.server.op.session.Session - New session established for 7cba99a7-6238-42a3-871b-c14a2131ae6c
324815 [gremlin-server-worker-1] ERROR org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager - Could not create GremlinScriptEngine for gremlin-groovy
java.lang.IllegalStateException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script6
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.lambda$createGremlinScriptEngine$16(DefaultGremlinScriptEngineManager.java:464)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.createGremlinScriptEngine(DefaultGremlinScriptEngineManager.java:450)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.getEngineByName(DefaultGremlinScriptEngineManager.java:219)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.lambda$getEngineByName$0(CachedGremlinScriptEngineManager.java:57)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.getEngineByName(CachedGremlinScriptEngineManager.java:57)
at org.apache.tinkerpop.gremlin.server.op.session.Session.registerMetrics(Session.java:258)
at java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:559)
at org.apache.tinkerpop.gremlin.server.op.session.Session.
It seems to work. I used the HBase 2 API. The HBase 1 API didn't work. I can see it connecting to BigTable and creating the table. But I am not able to create a vertex. I get the following error.
gremlin> :remote connect tinkerpop.server conf/remote.yaml session ==>Configured localhost/127.0.0.1:8182-[b05eb450-c049-4cf4-9056-ce6b4bba4d35] gremlin> :remote console ==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182]-[b05eb450-c049-4cf4-9056-ce6b4bba4d35] - type ':remote console' to return to local mode gremlin> gremlin> v1 = graph.addVertex( label, "hello") gremlin-groovy is not an available GremlinScriptEngine Type ':help' or ':h' for help. Display stack trace? [yN] gremlin>
Ivan,
Could you share how you deployed to K8s?
Thanks,
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Korjavin Ivan notifications@github.com Sent: Tuesday, March 31, 2020 3:27:46 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
storage.backend: hbase storage.directory: null storage.hbase.ext.google.bigtable.instance.id: {realid} storage.hbase.ext.google.bigtable.project.id: {realproject} storage.hbase.ext.hbase.client.connection.impl: com.google.cloud.bigtable.hbase2_x.BigtableConnection index.search.backend: elasticsearch index.search.directory: null index.search.hostname: 192.168.1.69 cache.db-cache: true cache.db-cache-clean-wait: 20 cache.db-cache-time: 180000 cache.db-cache-size: 0.5 ids.authority.wait-time: 0
gremlin-server.yaml
host: 0.0.0.0 port: 8182 scriptEvaluationTimeout: 30000 channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer graphs: { graph: /etc/opt/janusgraph/janusgraph.properties } scriptEngines: { gremlin-groovy: { plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {}, org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {}, org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {}, org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]}, org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/empty-sample.groovy]}}}} serializers: ....
After start it looks good:
2718 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com 2962 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtableadmin.googleapis.com 46138 [main] INFO org.janusgraph.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 13 OK out of 13 attempts in PT0.068S 54622 [main] INFO org.janusgraph.graphdb.idmanagement.UniqueInstanceIdRetriever - Generated unique-instance-id=ac11000330-9c77409ef4e31 54655 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection 54655 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting google.bigtable.project.id=... 54655 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting google.bigtable.instance.id=... 54657 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Opening session for projectId phonic-heaven-126923, instanceId live3pm, on data host bigtable.googleapis.com, admin host bigtableadmin.googleapis.com. 54657 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Bigtable options: BigtableOptions{dataHost=bigtable.googleapis.com, adminHost=bigtableadmin.googleapis.com, port=443, projectId=phonic-heaven-126923, instanceId=..., appProfileId=, userAgent=hbase-2.1.5, credentialType=DefaultCredentials, dataChannelCount=32, retryOptions=RetryOptions{retriesEnabled=true, allowRetriesWithoutTimestamp=false, statusToRetryOn=[ABORTED, UNAUTHENTICATED, DEADLINE_EXCEEDED, UNAVAILABLE], initialBackoffMillis=5, maxElapsedBackoffMillis=60000, backoffMultiplier=1.5, streamingBufferSize=60, readPartialRowTimeoutMillis=60000, maxScanTimeoutRetries=3}, bulkOptions=BulkOptions{asyncMutatorCount=2, useBulkApi=true, bulkMaxKeyCount=125, bulkMaxRequestSize=1048576, autoflushMs=0, maxInflightRpcs=320, maxMemory=47762636, enableBulkMutationThrottling=false, bulkMutationRpcTargetMs=100}, callOptionsConfig=CallOptionsConfig{useTimeout=false, shortRpcTimeoutMs=60000, longRpcTimeoutMs=600000, mutateRpcTimeoutMs=600000, readStreamRpcTimeoutMs=600000}, usePlaintextNegotiation=false, useCachedDataPool=false, useBatch=false, useGCJClient=false}
Then I connect with gremlin
gremlin> :remote connect tinkerpop.server conf/remote.yaml session ==>Configured localhost/127.0.0.1:8182-[7cba99a7-6238-42a3-871b-c14a2131ae6c] gremlin> :remote console ==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182]-[7cba99a7-6238-42a3-871b-c14a2131ae6c] - type ':remote console' to return to local mode gremlin> GraphOfTheGodsFactory.load(graph)
And logs:
322524 [gremlin-server-worker-1] INFO org.apache.tinkerpop.gremlin.server.op.session.Session - New session established for 7cba99a7-6238-42a3-871b-c14a2131ae6c
324815 [gremlin-server-worker-1] ERROR org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager - Could not create GremlinScriptEngine for gremlin-groovy
java.lang.IllegalStateException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script6
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.lambda$createGremlinScriptEngine$16(DefaultGremlinScriptEngineManager.java:464)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.createGremlinScriptEngine(DefaultGremlinScriptEngineManager.java:450)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.getEngineByName(DefaultGremlinScriptEngineManager.java:219)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.lambda$getEngineByName$0(CachedGremlinScriptEngineManager.java:57)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.getEngineByName(CachedGremlinScriptEngineManager.java:57)
at org.apache.tinkerpop.gremlin.server.op.session.Session.registerMetrics(Session.java:258)
at java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:559)
at org.apache.tinkerpop.gremlin.server.op.session.Session.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-606856168, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVEIGYC3OMUPGKQOB3LRKJG4FANCNFSM4LUI25EQ.
korjavin
commented
4 years ago Yes, it creates tables, but then nothing.
I tested it in local docker, and also in google cloud.
I deploy with helm chart.
haroon-qureshi
commented
4 years ago Can you share your helm chart?
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Wednesday, April 1, 2020 4:37 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Yes, it creates tables, but then nothing.
I tested it in local docker, and also in google cloud.
I deploy with helm chart.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-607502398, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVFTDXHOWRPVFPZBNYTRKOXYLANCNFSM4LUI25EQ.
haroon-qureshi
commented
4 years ago I'm getting the same script engine manager error that you are.
M: +1.312.927.0276
From: Haroon Qureshi qureshih@outlook.com Sent: Wednesday, April 1, 2020 4:38 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com; JanusGraph/janusgraph reply@reply.github.com Cc: Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Can you share your helm chart?
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Wednesday, April 1, 2020 4:37 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Yes, it creates tables, but then nothing.
I tested it in local docker, and also in google cloud.
I deploy with helm chart.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-607502398, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVFTDXHOWRPVFPZBNYTRKOXYLANCNFSM4LUI25EQ.
korjavin
commented
4 years ago I took this one https://hub.helm.sh/charts/stable/janusgraph
and updated it.
haroon-qureshi
commented
4 years ago Ok ... I did the same thing 🙂
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Wednesday, April 1, 2020 4:40 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
I took this one https://hub.helm.sh/charts/stable/janusgraph
and updated it.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-607503539, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVFPAFZXDLITB6IAAE3RKOYDRANCNFSM4LUI25EQ.
korjavin
commented
4 years ago Any luck?
haroon-qureshi
commented
4 years ago I saw this note, which seems to work. But now I'm getting errors stating that my ElasticSearch needs to be upgraded. So i'm going to do that now and hopefully everything will work.
https://github.com/JanusGraph/janusgraph/issues/1308
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Thursday, April 2, 2020 4:09 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Any luck?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-607720862, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVHSZQDZ43XW3FNPMJDRKRI3ZANCNFSM4LUI25EQ.
haroon-qureshi
commented
4 years ago Ivan
Were you able to get this to work? I’m having issues with elastic search version at the moment.
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Haroon Qureshi qureshih@outlook.com Sent: Thursday, April 2, 2020 1:06:59 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com; JanusGraph/janusgraph reply@reply.github.com Cc: Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
I saw this note, which seems to work. But now I'm getting errors stating that my ElasticSearch needs to be upgraded. So i'm going to do that now and hopefully everything will work.
https://github.com/JanusGraph/janusgraph/issues/1308
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Thursday, April 2, 2020 4:09 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Any luck?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-607720862, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVHSZQDZ43XW3FNPMJDRKRI3ZANCNFSM4LUI25EQ.
korjavin
commented
4 years ago I haven't noticed problem with elastic. But bigtable doesn't work for me.
haroon-qureshi
commented
4 years ago Big table isn’t working for me either. It connects but I can’t create any vertices in it. When I look in the logs it says it can’t instantiate the janusgraph because it can’t connect to elastic search. The connection times out. I’m hoping that once that is fixed it can instantiate the instance and be able to work. I’m trying to debug why it is timing out.
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Korjavin Ivan notifications@github.com Sent: Monday, April 6, 2020 10:55:55 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
I haven't noticed problem with elastic. But bigtable doesn't work for me.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-609880609, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVF4GLXGDGA6UJJOT3DRLH3QXANCNFSM4LUI25EQ.
korjavin
commented
4 years ago Can you show me logs?
haroon-qureshi
commented
4 years ago sure ..
5350 [main] WARN org.janusgraph.diskstorage.es.rest.RestElasticSearchClient - Unable to determine Elasticsearch server version. Default to SEVEN.
java.net.ConnectException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.249.73:9200]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:823)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.getMajorVersion(RestElasticSearchClient.java:137)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Monday, April 6, 2020 11:02 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Can you show me logs?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-609884498, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVEDQQB6AG3FNUT74ETRLH4JXANCNFSM4LUI25EQ.
haroon-qureshi
commented
4 years ago gremlin> :remote connect tinkerpop.server conf/remote.yaml session ==>Configured localhost/127.0.0.1:8182-[966ea680-d4c3-4db3-8739-1b87c2ae812b] gremlin> :remote console ==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182]-[966ea680-d4c3-4db3-8739-1b87c2ae812b] - type ':remote console' to return to local mode gremlin> v1 = graph.addVertex( label, "hello") No such property: graph for class: Script2 Type ':help' or ':h' for help. Display stack trace? [yN][A gremlin> v1 = graph.addVertex("hello") No such property: graph for class: Script3 Type ':help' or ':h' for help. Display stack trace? [yN][A gremlin> v1 = graph.addV('hello') No such property: graph for class: Script4 Type ':help' or ':h' for help. Display stack trace? [yN] gremlin> :q!
M: +1.312.927.0276
From: Haroon Qureshi qureshih@outlook.com Sent: Monday, April 6, 2020 11:12 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com; JanusGraph/janusgraph reply@reply.github.com Cc: Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
sure ..
5350 [main] WARN org.janusgraph.diskstorage.es.rest.RestElasticSearchClient - Unable to determine Elasticsearch server version. Default to SEVEN.
java.net.ConnectException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.249.73:9200]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:823)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.getMajorVersion(RestElasticSearchClient.java:137)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.
M: +1.312.927.0276
From: Korjavin Ivan notifications@github.com Sent: Monday, April 6, 2020 11:02 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Can you show me logs?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-609884498, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVEDQQB6AG3FNUT74ETRLH4JXANCNFSM4LUI25EQ.
korjavin
commented
4 years ago I'll try again and check a bit later
porunov
commented
4 years ago I didn't check it but some of the next tips might help:
1) In 0.5.x only ElasticSearch 6 and 7 works. If you are tuning a lower version, upgrade your ElasticSearch. Also, if you are upgrading to ElasticSearch 7 from ElasticSearch 6 AND you are not reindexing you ElasticSearch index to use indexes without mapping types, you must set use-mapping-for-es7 parameter to true (I.e. index.[X].elasticsearch.use-mapping-for-es7 = true). (See upgrade instruction: https://docs.janusgraph.org/changelog/#elasticsearch-upgrade-from-660-to-761-and-drop-support-for-5x-version).
2) gremlin-server had small bugs and was fixed 2 days ago. Re-download 0.5.1 artifacts before using gremlin-server.
3) Notice, gremlin server uses inmemory backend and no mixed index backend by default. Make sure you configure it to use hbase and elasticsearch.
haroon-qureshi
commented
4 years ago Thanks. We are using version 6.8.6 of elasticaearch and have done the other points as well and still having this issue. I think it’s a cluster/chart issue since I can’t curl the ES client IP address either.
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Oleksandr Porunov notifications@github.com Sent: Monday, April 6, 2020 11:41:43 AM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
I didn't check it but some of the next tips might help:
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-609905966, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVDUSYNC233FRWBNSOLRLIA4PANCNFSM4LUI25EQ.
porunov
commented
4 years ago JanusGraph 0.5.1 is released. JanusGraph 0.5.1 is uploaded into docker hub.
korjavin
commented
4 years ago Looks like it started working for me
haroon-qureshi
commented
4 years ago Oh did you reinstall? Can you share what modifications you made to your helm chart?
Haroon A. Qureshi — M: +1.312.927.0276 | E: qureshih@outlook.com
From: Korjavin Ivan notifications@github.com Sent: Wednesday, April 8, 2020 3:15:47 PM To: JanusGraph/janusgraph janusgraph@noreply.github.com Cc: Haroon A. Qureshi qureshih@outlook.com; Mention mention@noreply.github.com Subject: Re: [JanusGraph/janusgraph] JanusGraph/Bigtable integration failing with current Bigtable version (#2060)
Looks like it started working for me
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/2060#issuecomment-611170660, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFIKCVE3EFUJTWT2EBRAUEDRLTLPHANCNFSM4LUI25EQ.
korjavin
commented
4 years ago I checked it with local docker image. I will check all those soon.
Just now, I run into a problem again. It works for simple cases. When I tried to do more inserts, I got
579927 [gremlin-server-session-1] WARN org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker - Lock write succeeded but took too long: duration PT0.214S exceeded limit PT0.1S
580094 [gremlin-server-session-1] WARN org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker - Lock write succeeded but took too long: duration PT0.167S exceeded limit PT0.1S
580269 [gremlin-server-session-1] WARN org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker - Lock write succeeded but took too long: duration PT0.175S exceeded limit PT0.1S
580445 [gremlin-server-session-1] ERROR org.janusgraph.graphdb.database.StandardJanusGraph - Could not commit transaction [1] due to exception
org.janusgraph.diskstorage.locking.TemporaryLockingException: Temporary locking failure
at org.janusgraph.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:309)on GraphOfTheGodsFactory.load(graph)
korjavin
commented
4 years ago @porunov Is there something that I should tweak ?
korjavin
commented
4 years ago @haroon-qureshi
look here: https://github.com/korjavin/docker-janusgraph
I provided readme with details there
porunov
commented
4 years ago @korjavin
Try to add storage.batch-loading=true to your https://github.com/korjavin/docker-janusgraph/blob/master/janusgraph-hbase.properties to disable consistency checks by default or disable them explicitly per transaction (consistencyChecks(false)) https://docs.janusgraph.org/basics/transactions/
P.S. don't forget to drop your uploaded graph before loading GraphOfTheGodsFactory.load(graph) again (i.e. ConfiguredGraphFactory.drop("graphName") or JanusGraphFactory.drop(graph)).
korjavin
commented
4 years ago I made a fresh install. I did JanusGraphFactory.drop(graph). Deleted bigtable table and added the option you suggested.
It started well:
1253 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
1414 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtableadmin.googleapis.com
63366 [main] INFO org.janusgraph.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 13 OK out of 13 attempts in PT0.034S
87878 [main] INFO org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder - Set timestamps to MILLI according to storage backend preference
88408 [main] INFO org.janusgraph.graphdb.idmanagement.UniqueInstanceIdRetriever - Generated unique-instance-id=ac11000330-9ffcc893f5d41
88428 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
88428 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting google.bigtable.project.id=phonic-heaven-126923
88428 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting google.bigtable.instance.id=live3pm
88429 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Opening session for projectId phonic-heaven-126923, instanceId live3pm, on data host bigtable.googleapis.com, admin host bigtableadmin.googleapis.com.
88429 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Bigtable options: BigtableOptions{dataHost=bigtable.googleapis.com, adminHost=bigtableadmin.googleapis.com, port=443, projectId=phonic-heaven-126923, instanceId=live3pm, appProfileId=, userAgent=hbase-2.1.5, credentialType=DefaultCredentials, dataChannelCount=32, retryOptions=RetryOptions{retriesEnabled=true, allowRetriesWithoutTimestamp=false, statusToRetryOn=[UNAUTHENTICATED, DEADLINE_EXCEEDED, ABORTED, UNAVAILABLE], initialBackoffMillis=5, maxElapsedBackoffMillis=60000, backoffMultiplier=1.5, streamingBufferSize=60, readPartialRowTimeoutMillis=60000, maxScanTimeoutRetries=3}, bulkOptions=BulkOptions{asyncMutatorCount=2, useBulkApi=true, bulkMaxKeyCount=125, bulkMaxRequestSize=1048576, autoflushMs=0, maxInflightRpcs=320, maxMemory=47762636, enableBulkMutationThrottling=false, bulkMutationRpcTargetMs=100}, callOptionsConfig=CallOptionsConfig{useTimeout=false, shortRpcTimeoutMs=60000, longRpcTimeoutMs=600000, mutateRpcTimeoutMs=600000, readStreamRpcTimeoutMs=600000}, usePlaintextNegotiation=false, useCachedDataPool=false, useBatch=false, useGCJClient=false}.
88429 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88431 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88432 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88434 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88435 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88436 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88438 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88439 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88440 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88441 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88442 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88443 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88444 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88445 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88447 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88448 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88449 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88451 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88452 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88453 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88454 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88455 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88456 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88457 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88458 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88459 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88460 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88461 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88464 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88466 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88468 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88471 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtable.googleapis.com
88474 [main] INFO com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession - Creating new channel for bigtableadmin.googleapis.com
88481 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
91137 [main] INFO org.janusgraph.diskstorage.Backend - Initiated backend operations thread pool of size 16
92456 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
95487 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.
96941 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
133682 [main] INFO org.janusgraph.graphdb.database.IndexSerializer - Hashing index keys
134252 [main] INFO org.janusgraph.diskstorage.log.kcvs.KCVSLog - Loaded unidentified ReadMarker start time 2020-04-08T20:44:00.125Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@32456db0
134253 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] was successfully configured via [/etc/opt/janusgraph/janusgraph.properties].
134253 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
134295 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized GremlinExecutor and preparing GremlinScriptEngines instances.
136278 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized gremlin-groovy GremlinScriptEngine and registered metrics
136282 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - A GraphTraversalSource is now bound to [g] with graphtraversalsource[standardjanusgraph[hbase:[127.0.0.1]], standard]
136292 [main] INFO org.apache.tinkerpop.gremlin.server.op.OpLoader - Adding the standard OpProcessor.
136294 [main] INFO org.apache.tinkerpop.gremlin.server.op.OpLoader - Adding the session OpProcessor.
136376 [main] INFO org.apache.tinkerpop.gremlin.server.op.OpLoader - Adding the traversal OpProcessor.
136385 [main] INFO org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor - Initialized cache for TraversalOpProcessor with size 1000 and expiration time of 600000 ms
136431 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Executing start up LifeCycleHook
136449 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Executed once at startup of Gremlin Server.
136461 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - idleConnectionTimeout was set to 0 which resolves to 0 seconds when configuring this value - this feature will be disabled
136461 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - keepAliveInterval was set to 0 which resolves to 0 seconds when configuring this value - this feature will be disabled
136465 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0 serialization class is deprecated.
136534 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v3.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0
136534 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0 serialization class is deprecated.
136534 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v3.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0
136556 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v3.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0
136556 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0
136560 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0 serialization class is deprecated.
136562 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
136565 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0 serialization class is deprecated.
136566 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-lite with org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0
136567 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0 serialization class is deprecated.
136567 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
136568 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - The org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0 serialization class is deprecated.
136574 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v2.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0
136574 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - application/json already has org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0 configured - it will not be replaced by org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, change order of serialization configuration if this is not desired.
136585 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0
136587 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - application/json already has org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0 configured - it will not be replaced by org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, change order of serialization configuration if this is not desired.
136656 [gremlin-server-boss-1] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
136656 [gremlin-server-boss-1] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Channel started at port 8182.GraphOfTheGodsFactory.load(graph)
causes:
263260 [gremlin-server-session-1] ERROR com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation - Could not complete RPC. Failure #0, got: Status{code=UNKNOWN, description=null, cause=java.util.concurrent.RejectedExecutionException: Task com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor@afa5396 rejected from java.util.concurrent.ThreadPoolExecutor@559d81c0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 331]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:93)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:86)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:755)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:705)
at com.google.bigtable.repackaged.io.grpc.internal.FailingClientStream.start(FailingClientStream.java:54)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:287)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:210)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor$1.checkedStart(HeaderInterceptor.java:58)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor$1.checkedStart(HeaderInterceptor.java:58)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:87)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.BigtableAsyncUtilities$Default$1.start(BigtableAsyncUtilities.java:89)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.CallController.setCallAndStart(CallController.java:71)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.run(AbstractRetryingOperation.java:377)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getAsyncResult(AbstractRetryingOperation.java:437)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:448)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableTableAdminGrpcClient.listTables(BigtableTableAdminGrpcClient.java:185)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableTableAdminClientWrapper.listTables(BigtableTableAdminClientWrapper.java:126)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:274)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:237)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:226)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.tableExists(AbstractBigtableAdmin.java:140)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:110)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:726)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:821)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:487)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:218)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:334)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:325)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:79)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:65)
at org.janusgraph.example.GraphOfTheGodsFactory$load.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at Script5.run(Script5.groovy:1)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:674)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
} on channel null.
Trailers: Metadata()
java.util.concurrent.RejectedExecutionException: Task com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor@afa5396 rejected from java.util.concurrent.ThreadPoolExecutor@559d81c0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 331]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:93)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:86)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:755)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:705)
at com.google.bigtable.repackaged.io.grpc.internal.FailingClientStream.start(FailingClientStream.java:54)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:287)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:210)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor$1.checkedStart(HeaderInterceptor.java:58)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor$1.checkedStart(HeaderInterceptor.java:58)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:87)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.BigtableAsyncUtilities$Default$1.start(BigtableAsyncUtilities.java:89)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.CallController.setCallAndStart(CallController.java:71)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.run(AbstractRetryingOperation.java:377)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getAsyncResult(AbstractRetryingOperation.java:437)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:448)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableTableAdminGrpcClient.listTables(BigtableTableAdminGrpcClient.java:185)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableTableAdminClientWrapper.listTables(BigtableTableAdminClientWrapper.java:126)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:274)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:237)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:226)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.tableExists(AbstractBigtableAdmin.java:140)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:110)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:726)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:821)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:487)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:218)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:334)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:325)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:79)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:65)
at org.janusgraph.example.GraphOfTheGodsFactory$load.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at Script5.run(Script5.groovy:1)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:674)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
263263 [gremlin-server-session-1] WARN org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor - Exception processing a script on request [RequestMessage{, requestId=d44638b7-9c83-46e3-bbfe-956f33823e55, op='eval', processor='session', args={gremlin=GraphOfTheGodsFactory.load(graph), session=6e761ad9-0972-4cae-80a6-17a07d20b9bd, manageTransaction=false, userAgent=Gremlin Console/3.4.6, batchSize=64}}].
java.util.concurrent.RejectedExecutionException: Task com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor@afa5396 rejected from java.util.concurrent.ThreadPoolExecutor@559d81c0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 331]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:93)
at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:86)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:755)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:705)
at com.google.bigtable.repackaged.io.grpc.internal.FailingClientStream.start(FailingClientStream.java:54)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:287)
at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:210)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor$1.checkedStart(HeaderInterceptor.java:58)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor$1.checkedStart(HeaderInterceptor.java:58)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:87)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:231)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.BigtableAsyncUtilities$Default$1.start(BigtableAsyncUtilities.java:89)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.CallController.setCallAndStart(CallController.java:71)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.run(AbstractRetryingOperation.java:377)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getAsyncResult(AbstractRetryingOperation.java:437)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:448)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableTableAdminGrpcClient.listTables(BigtableTableAdminGrpcClient.java:185)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableTableAdminClientWrapper.listTables(BigtableTableAdminClientWrapper.java:126)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:274)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:237)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.listTableNames(AbstractBigtableAdmin.java:226)
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.tableExists(AbstractBigtableAdmin.java:140)
at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:110)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:726)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:821)
at org.janusgraph.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:487)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:218)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:334)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:325)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:79)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:65)
at org.janusgraph.example.GraphOfTheGodsFactory$load.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at Script5.run(Script5.groovy:1)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:674)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
269166 [gremlin-server-session-1] WARN org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor - Exception processing a script on request [RequestMessage{, requestId=e98647c7-abf7-4bb9-af9d-2d1cb68b172a, op='eval', processor='session', args={gremlin=GraphOfTheGodsFactory.load(graph), session=6e761ad9-0972-4cae-80a6-17a07d20b9bd, manageTransaction=false, userAgent=Gremlin Console/3.4.6, batchSize=64}}].
java.lang.IllegalStateException: Cannot perform operations on a closed connection
at com.google.bigtable.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:511)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.ChannelPool.newCall(ChannelPool.java:282)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor.interceptCall(HeaderInterceptor.java:54)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.HeaderInterceptor.interceptCall(HeaderInterceptor.java:54)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.WatchdogInterceptor.interceptCall(WatchdogInterceptor.java:45)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.bigtable.repackaged.io.grpc.auth.ClientAuthInterceptor.interceptCall(ClientAuthInterceptor.java:65)
at com.google.bigtable.repackaged.io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.BigtableAsyncUtilities$Default$1.newCall(BigtableAsyncUtilities.java:80)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.CallController.setCallAndStart(CallController.java:70)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.run(AbstractRetryingOperation.java:377)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.RetryingStreamOperation.run(RetryingStreamOperation.java:65)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getAsyncResult(AbstractRetryingOperation.java:437)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:448)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.readFlatRowsList(BigtableDataGrpcClient.java:386)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableDataClientWrapper.readFlatRowsList(BigtableDataClientWrapper.java:254)
at com.google.cloud.bigtable.hbase.AbstractBigtableTable.getResults(AbstractBigtableTable.java:264)
at com.google.cloud.bigtable.hbase.AbstractBigtableTable.get(AbstractBigtableTable.java:258)
at com.google.cloud.bigtable.hbase.AbstractBigtableTable.get(AbstractBigtableTable.java:235)
at org.janusgraph.diskstorage.hbase.HTable1_0.get(HTable1_0.java:45)
at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore.getHelper(HBaseKeyColumnValueStore.java:172)
at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore.getSlice(HBaseKeyColumnValueStore.java:92)
at org.janusgraph.diskstorage.keycolumnvalue.KCVSUtil.get(KCVSUtil.java:55)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$2.call(KCVSLog.java:829)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$2.call(KCVSLog.java:826)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog.readSetting(KCVSLog.java:826)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog.<init>(KCVSLog.java:269)
at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:218)
at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:334)
at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:325)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:79)
at org.janusgraph.example.GraphOfTheGodsFactory.load(GraphOfTheGodsFactory.java:65)
at org.janusgraph.example.GraphOfTheGodsFactory$load.call(Unknown Source)
at Script5.run(Script5.groovy:1)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:674)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Another question.
I'm getting this:
23920 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
25711 [main] INFO org.janusgraph.diskstorage.Backend - Initiated backend operations thread pool of size 16
28104 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
31123 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.
32588 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
35400 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.
36863 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
39338 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.
40790 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
43879 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.
45333 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
48290 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.
49749 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was disabled in memory only.
52629 [main] WARN com.google.cloud.bigtable.hbase2_x.BigtableAdmin - Table janusgraph was enabled in memory only.without querying janusgraph. Just after start for a long time.
Please post all questions and issues on janusgraph-users before opening a GitHub issue. Your questions will reach a wider audience there, and if we confirm that there is a bug, then you can open a new issue.
Please include configurations and logs if available.
For confirmed bugs, please report:
When trying to load my .properties file (
graph = JanusGraphFactory.open('conf/test.properties')), I am also seeing the error mentioned in the mailing list thread (below). My properties file is identical to the suggested changes to work with Bigtable here: https://docs.janusgraph.org/storage-backend/bigtable/Stack Trace (if you have one)
This is happening because in the last part of the stacktrace, the Bigtable Hbase 1.x client does not have a constructor mapping to the parameters listed which are called reflectively. (
java.lang.NoSuchMethodException: com.google.cloud.bigtable.hbase1_x.BigtableConnection.<init>(org.apache.hadoop.conf.Configuration, java.util.concurrent.ExecutorService, org.apache.hadoop.hbase.security.User))To fix this, I'd like to propose upgrading the
bigtable-hbase-1.x-shadeddependency to thebigtable-hbase-2.x-shadedversion, which is compatible with the version of hbase and will not throw this error. I will create a PR for this unless there are any other ideas or objections. Thanks!