search

davebshow / aiogremlin

http://aiogremlin.readthedocs.org/en/latest/
Other
46 stars 13 forks source link

Serialization problem #2

Closed spinus closed 9 years ago

spinus commented 9 years ago

My setup is: titan-0.9.0-M2-hadoop1 (berkleyDB backend, no index). When I connect to gremlin server using gremlin shell I can do stuff:

gremlin> root@a33e29cbee9a:/opt/titan# ./bin/gremlin.sh 

         \,,,/
         (o o)
-----oOOo-(3)-oOOo-----
plugin activated: tinkerpop.server
plugin activated: tinkerpop.utilities
19:04:52 INFO  org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph  - HADOOP_GREMLIN_LIBS is set to: /opt/titan-0.9.0-M2-hadoop1/bin/../lib
plugin activated: tinkerpop.hadoop
plugin activated: tinkerpop.tinkergraph
plugin activated: aurelius.titan
gremlin> :remote connect tinkerpop.server conf/remote.yaml
==>Connected - localhost/127.0.0.1:8182
gremlin> :> g
==>standardtitangraph[berkeleyje:/data/]
gremlin> :> g.addVertex("name","stephen")
==>v[4344]

But with aiogremlin with the code

    resp = yield from client.submit('''
g
    ''')

I receive the error:

  File "/usr/lib/python3.4/asyncio/tasks.py", line 287, in _wakeup
    value = future.result()
  File "/usr/lib/python3.4/asyncio/futures.py", line 276, in result
    raise self._exception
  File "/home/spinus/lib/python3.4/site-packages/aiohttp/parsers.py", line 131, in feed_data
    self._parser.send(data)
  File "/home/spinus/lib/python3.4/site-packages/aiogremlin/subprotocol.py", line 41, in gremlin_response_parser
    raise GremlinServerError(message.status_code, message.message)
aiogremlin.exceptions.GremlinServerError: Code [599]: SERIALIZATION. The server was not capable of serializing an object that was returned from the script supplied on the requst. Either transform the object into something Gremlin Server can process within the script or install mapper serialization classes to Gremlin Server..{}
2015-08-07 20:08:55,260 - asyncio - ERROR - Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f708111f908>
2015-08-07 20:08:55,260 - asyncio - ERROR - Unclosed response
client_response: <ClientResponse(ws://localhost:8182/) [101 Switching Protocols]>
<_CIMultiDictProxy {'UPGRADE': 'websocket', 'CONNECTION': 'Upgrade', 'SEC-WEBSOCKET-ACCEPT': 'R8h30JRgaMI63ckjhMU2WpvmYC4='}>

On gremlin server errors look like:

9309867 [gremlin-server-worker-1] WARN  org.apache.tinkerpop.gremlin.server.handler.WsGremlinBinaryRequestDecoder  - Gremlin Server is not configured with a serializer for the requested mime type [application/json] - using org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 by default
9309869 [gremlin-server-worker-1] WARN  org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0  - Response [ResponseMessage{requestId=e83b8d8a-4d34-431a-9478-b0327552c407, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[standardtitangraph[berkeleyje:/data/]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
9309869 [gremlin-server-worker-1] WARN  org.apache.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder  - The result [ResponseResult{data=[standardtitangraph[berkeleyje:/data/]], meta={}}] in the request e83b8d8a-4d34-431a-9478-b0327552c407 could not be serialized and returned.
org.apache.tinkerpop.gremlin.driver.ser.SerializationException: com.fasterxml.jackson.databind.JsonMappingException: Cannot get namespace of root (through reference chain: java.util.ArrayList[0]->com.thinkaurelius.titan.graphdb.database.StandardTitanGraph["backend"]->com.thinkaurelius.titan.diskstorage.Backend["storeManager"]->com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter["features"]->com.thinkaurelius.titan.diskstorage.keycolumnvalue.StandardStoreFeatures["keyConsistentTxConfig"]->com.thinkaurelius.titan.diskstorage.configuration.ModifiableConfiguration["rootNamespace"]->com.thinkaurelius.titan.diskstorage.configuration.ConfigNamespace["namespace"])
    at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.serializeResponseAsBinary(AbstractGraphSONMessageSerializerV1d0.java:116)
    at org.apache.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder.encode(WsGremlinResponseEncoder.java:65)
    at org.apache.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder.encode(WsGremlinResponseEncoder.java:45)
    at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:691)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:626)
    at org.apache.tinkerpop.gremlin.server.handler.IteratorHandler.write(IteratorHandler.java:119)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
    at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
    at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Cannot get namespace of root (through reference chain: java.util.ArrayList[0]->com.thinkaurelius.titan.graphdb.database.StandardTitanGraph["backend"]->com.thinkaurelius.titan.diskstorage.Backend["storeManager"]->com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter["features"]->com.thinkaurelius.titan.diskstorage.keycolumnvalue.StandardStoreFeatures["keyConsistentTxConfig"]->com.thinkaurelius.titan.diskstorage.configuration.ModifiableConfiguration["rootNamespace"]->com.thinkaurelius.titan.diskstorage.configuration.ConfigNamespace["namespace"])
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232)
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:197)
    at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:186)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:640)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:541)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:632)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:541)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:632)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:541)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:632)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:541)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:632)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:541)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:632)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21)
    at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)
    at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:49)
    at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:235)
    at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:205)
    at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:197)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:114)
    at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:2811)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:2292)
    at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.serializeResponseAsBinary(AbstractGraphSONMessageSerializerV1d0.java:107)
    ... 16 more
Caused by: java.lang.IllegalArgumentException: Cannot get namespace of root
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93)
    at com.thinkaurelius.titan.diskstorage.configuration.ConfigElement.getNamespace(ConfigElement.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.get(BeanPropertyWriter.java:726)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:506)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:632)
    ... 43 more

I'm not sure this is misconfiguration of server or gremlin serialization bug, have any idea?

Moreover, those lines:

2015-08-07 20:08:55,260 - asyncio - ERROR - Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f708111f908>
2015-08-07 20:08:55,260 - asyncio - ERROR - Unclosed response
client_response: <ClientResponse(ws://localhost:8182/) [101 Switching Protocols]>
<_CIMultiDictProxy {'UPGRADE': 'websocket', 'CONNECTION': 'Upgrade', 'SEC-WEBSOCKET-ACCEPT': 'R8h30JRgaMI63ckjhMU2WpvmYC4='}>

appear on every request, even when gremlin script is evaluated correctly and there is no serialization problems (for example when script is 2+2).

davebshow commented 9 years ago

I'm not 100% sure, but it may have to do with the Titan conf file. In titan/conf/gremlin-server/gremlin-server.yaml make sure that channelizer is set to WebSocketChannelizer like such:

channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer

Let me know if that fixes your problem.

Cheers,

Dave

davebshow commented 9 years ago

Looking a bit more, I don't think this is the problem. I'll take a closer look at this tomorrow morning and get back to you.

spinus commented 9 years ago

@davebshow sorry I didnt include configs. Here they are: titan.properties:

gremlin.graph=com.thinkaurelius.titan.core.TitanFactory
storage.backend=berkeleyje
storage.directory=/data/

gremlin.yaml:

host: 0.0.0.0
port: 8182
threadPoolWorker: 1
gremlinPool: 8
scriptEvaluationTimeout: 30000
serializedResponseTimeout: 30000
channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
graphs: {
  g: /service/titan.properties}
plugins:
  - aurelius.titan
scriptEngines: {
  gremlin-groovy: {
    imports: [java.lang.Math],
    staticImports: [java.lang.Math.PI],
    scripts: [scripts/empty-sample.groovy]},
  nashorn: {
      imports: [java.lang.Math],
      staticImports: [java.lang.Math.PI]}}
serializers:
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { useMapperFromGraph: graph }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { useMapperFromGraph: graph }}
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { useMapperFromGraph: graph }}
processors:
  - { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
metrics: {
  consoleReporter: {enabled: true, interval: 180000},
# csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
  jmxReporter: {enabled: true},
  slf4jReporter: {enabled: true, interval: 180000},
  gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
  graphiteReporter: {enabled: false, interval: 180000}}
threadPoolBoss: 1
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferHighWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
  enabled: false}
davebshow commented 9 years ago

Ok a couple things. I found the bug in that wasn't releasing the resource on error, and I fixed it in the most recent commit. But to be honest I don't think this is the source of your problems.

I noticed in your config that in graphs you have defined your graph name as g. This is fine in theory, but in the new gremlin the convention is to call this graph, because g is used for the TraversalSource object. This object is defined in the scripts file, and by default in scripts/empty-sample.groovy, g is defined as follows:

g = graph.traversal()

Maybe you have modified this, maybe not. However, this could be the source of your error, defining g in both conf/gremlin-server/gremlin-server.yaml and in scripts/empty-sample.groovy. What are the contents of the latter? Please check this and see if it solves your problem. I would first rename your graph to graph in the gremlin-server.yaml, and try:

resp = yield from client.submit('''graph'')

That should work. Then try:

resp = yield from client.submit('''g''')

And you should see the TraversalSource object. Let me know how this goes. If everything is good I will release a new version of aiogremlin with the bug fix.

Thanks @spinus!

davebshow commented 9 years ago

Also, please remember that you must explicitly close the client using the close coroutine when you are done:

yield from client.close()

I'll add this to the docs.

A way to get around this is by using the submit function.

spinus commented 9 years ago

Good stuff, it's moving forward.

empty-samply.groovy does exactly what you wrote. I didn't change anything.

I changed gremlin graph name to graph as you suggested. Didn't change anything else.

When I run submit("g") the response it:

[Message(status_code=200, data=[{'graph': {'present': True}, 'graphComputer': {'present': False}, 'strategies': [{'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$DecorationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$OptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$OptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$OptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$OptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$OptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$OptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$VendorOptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$VendorOptimizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$FinalizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$FinalizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$FinalizationStrategy'}, {'traversalCategory': 'org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy$VerificationStrategy'}]}], message={}, metadata='')]

It seems that traversal object works now. I can do query with g.V() and responses are ok.

When I use submit("graph") I still have serialization problem the same as I described before.

@davebshow Thank you for helping with this.

davebshow commented 9 years ago

Well I get that same graph serialization error using WebSockets and HTTP. I've posted a thread on Gremlin User's Google group. Let's see what Stephen Mallette has to say about it...

https://groups.google.com/forum/#!topic/gremlin-users/QdQ9PH0noxs

davebshow commented 9 years ago

Here I will quote the response from Stephen Mallette, who is the lead developer on the Gremlin Server. If you are having any other issues with unclosed clients or unreleased resources please open a separate issue. Cheers, Dave

"You will always get an error like that anytime you try to return a value that isn't understood as serializable given the serializer you are using. So it's just saying that it doesn't know what to do with "graph" which returns a Graph instance. Generally speaking, TinkerPop does not enforce any semantics regarding the serialization of a Graph object. It's a bit of a weird use case in a sense, especially a TitanGraph (i.e. billion edge graph), because it would just mean Gremlin Server would have to send back all the vertices/edges when it is sorta rare that you would ever want that. Not sure what the point of that would be. That said, I did think that serialization of a TinkerGraph would make sense as they are typically small and folks might want to ship around small subgraphs and such - so as of 3.1.x you will be able to do that."

spinus commented 9 years ago

@davebshow thank you for the effort.