Vertex Properties requests to Gremlin Server do not complete (Titan 1.0.0)

[adurrive]

Hello,

This issue is a follow up from https://github.com/jbmusso/gremlin-javascript/issues/19.

g.V().limit(1).properties()

Such requests do not complete and leave the client hanging. I have reproduced the issue with gremlin-javascript and aiogremlin (although not with the Gremlin console), using Titan 1.0.0 and the latest 1.0.1 build. The Gremlin server works as expected using TinkerGraph, which seems to indicate that the issue is related to Titan.

[spmallette]

When you used the Gremlin Console to :remote to Gremlin Server with Titan behind it, what configuration file did you pass to the :remote command?

[adurrive]

I have reproduced the issue locally with an out of the box Titan 1.0.0 using all default settings. Regarding the Gremlin console (which works) I used the default remote.yaml:

hosts: [localhost]
port: 8182
serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}

[spmallette]

just to be clear....are you saying that with that configuration you had success?

[adurrive]

Yes. The Gremlin console actually works perfectly. But using any other client (well, at least two of them), the above request does not complete: no response message is sent by the Gremlin server or received by the client.

Looking more closely at the logs, the Gremlin server always find the correct results but when the script returns vertex properties, it stops there leaving the client hanging.

[spmallette]

Your test with the console is fooling you a bit. You are using "text" serialization. serializeResultToString is true. That doesn't do an actual serialization of the object on the server. it just toStrings the result - so just about everything works when you do that. Can you try using conf/remote-objects.yaml instead and see if you get an error and if so, what that error is?

[adurrive]

You were spot on. The gremlin console without the "text" serialization hangs as well. Unfortunately I do not have any error, neither in the console nor the server logs. The console just times out.

[spmallette]

Ok - well at least we're narrowing things down. just to be clear again, does this only fail for Titan with that remote-objects.yaml or does it also fail for TinkerGraph?

[adurrive]

It fails only with Titan. TinkerGraph works well.

[spmallette]

took me a while to get to this, but i definitely see your problem. i don't clearly see what the problem is though unfortunately. will require further investigation. still looking....

[spmallette]

This bug has been a ride and I'm only at about 99% certainty that I have identified the root of the problem. Depending on how you look at it, it could be considered a "bug" in Gremlin Server or in Titan.

Here the failure, simplified:

gremlin> props = g.V().has('name').properties().toList()
16:02:58 WARN  com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>vp[name->marko]
==>vp[name->stephen]
gremlin> graph.tx().commit()
==>null
gremlin> graph.tx().isOpen()
==>false
gremlin> Thread.start { println props[1].id().toString() + " -  " +  props[1].value() }.join()
Exception in thread "Thread-19" java.lang.StackOverflowError

Basically, Gremlin Server is trying to serialize a Property outside of its original transaction in a separate thread. That's where it is probably a bug from the Gremlin Server side. On the other hand, this could be an inconsistency of Titan in that it does allow this to work for other graph elements - note that we don't have a problem with Vertex:

gremlin> vertices = g.V().has('name').toList()
16:12:45 WARN  com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>v[4240]
==>v[8392]
gremlin> Thread.start { println vertices[0].id() }.join()
4240
==>null
gremlin> graph.tx().commit()
==>null
gremlin> Thread.start { println vertices[0].id() }.join()
4240
==>null

Changes have been made for better consistency in "Transaction Management" in Gremlin Server on this issue:

https://issues.apache.org/jira/browse/TINKERPOP-1035

which should solve this problem on TinkerPop 3.1.1-incubating. It could be fixed on the Titan side, if the Property could be responsible for getting itself back into a transaction (the way Vertex did).

It's kinda dumpy that Gremlin Server swallowed the StackOverflow during serialization. Still not sure why that happened - it didn't show up anywhere which made this super hard to figure out.

Anywho - wow! :weary: :gun:

[adurrive]

Ouch, thanks for tracking this down! Great to know the next TinkerPop version should solve this.

[pariswd]

Found this nearly year-old thread while researching why our properties() calls were either not actually being carried out or were failing, and the property() method always fails.

We're running TItan 1.0 with TinkerPop 3.0.2. Is the only fix for this problem for us to upgrade TinkerPop? We're concerned about backward compatibility with our existing applications.

[spmallette]

There have been a mass of bug fixes for Gremlin Server and serialization on the TinkerPop side since 3.0.2. Give my comment, I'd say that upgrading is your only path to fixing it short of figuring out how to backport TINKERPOP-1035 to the 3.0.x line of code and then building your own custom version of TinkerPop. I'm not so sure a backport would even be easy to do in any way shape or form though - we didn't maintain the 3.0.x line for very long (as in the same way we maintained the 3.1.x line which is now heading for 3.1.6) so there were a lot of changes there. I'm not sure how many were "breaking" but you could review the changes of importance in the upgrade docs to see what it takes to bump to the newer line from 3.0.x:

http://tinkerpop.apache.org/docs/3.2.3/upgrade/#_tinkerpop_3_1_1

[pariswd]

Hi Stephen, thanks for your response. Our concern about upgrading is that Titan page is still on v1.0.0, which supports TP 3.0.x. So while we're open to upgrading TinkerPop, we don't see that Titan supports it. There's a pull request from 10 months ago to support it , but that isn't merged.

The other challenge is if we want to use DynamoDB as the storage, the plugin has to support the Titan version as well. So with all of this in mind we feel somewhat stuck on 1.0/3.0.x.

Given that the only problem we're experiencing is that the properties() method fails on an already existing vertex, can you suggest any workaround for that specific problem that might allow us to limp along until upgrades are officially made?

[spmallette]

You could probably force iterate the properties yourself. Either convert them into a List<Map> (or whatever kind of data structure makes sense - the main thing is to make sure it is serializable by Gremlin Server and that the objects you use no longer needs to reference the original transaction that spawned it) in the script you submit to the server or you could try to use the DetachFactory in your script and call detach() on each Property you intend to send back like:

g.V().properties().map(p -> DetachFactory.detach(p.get()))

I'm pretty sure either of those options would work based on my reading of my own comments on this issue, but obviously haven't given it a try myself. HTH

[pariswd]

Thanks for your help, Stephen. Your comments helped me find an indirect way to set properties on an already existing vertex which the server had been erroring out on when I tried using a direct call to properties(). In case anyone is interested...

g.V().has('name','myvertex').sideEffect{it.get().properties()[0].element().property('foo','bar')}.valueMap()