search

neo4j-contrib / java-rest-binding

Java Bindings for the Neo4J Server REST API, providing an implementation of GraphDatabaseService
Other
120 stars 72 forks source link

REST Cypher Query Engine cannot return large result rows (> about 8000) #13

Open ceefour opened 12 years ago

ceefour commented 12 years ago 
final RestAPI restApi = graphDb.getRestAPI();

log.info("Deleting existing Person nodes...");
RestIndex<Node> personIndex = graphDb.index().forNodes("berbatikPerson");
RestCypherQueryEngine cypher = new RestCypherQueryEngine(restApi);
for (List<Map<String, Object>> rows : Iterables.partition(cypher.query("START n=node:berbatikPerson('*:*') RETURN n", new HashMap<String, Object>()), 100)) {
    List<Long> nodeIds = Lists.transform(rows, new Function<Map<String, Object>, Long>() {
        @Override
        public Long apply(Map<String, Object> row) {
            Node node = (Node) row.get("n");
            Long nodeId = node.getId();
            log.debug("Deleting Person node {}", nodeId);
            for (Relationship rel : node.getRelationships()) {
                rel.delete();
            }
            node.delete();
            return nodeId;
        }
    });
    log.info("Deleted {} nodes: {}", nodeIds.size(), nodeIds);
}

Result:

22:19:16 [main] INFO  i.c.b.m.a.RecreatePersonNodesFromLdap - Deleting existing Person nodes...
Exception in thread "main" java.lang.RuntimeException: Error importing personlikeproduct
    at id.co.bippo.mall.admin.RecreatePersonNodesFromLdap.run(RecreatePersonNodesFromLdap.java:170)
    at id.co.bippo.mall.admin.RecreatePersonNodesFromLdap.main(RecreatePersonNodesFromLdap.java:178)
Caused by: java.lang.RuntimeException: org.codehaus.jackson.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@52639bf1; line: 1, column: 2]
    at org.neo4j.rest.graphdb.util.JsonHelper.readJson(JsonHelper.java:56)
    at org.neo4j.rest.graphdb.util.JsonHelper.jsonToMap(JsonHelper.java:43)
    at org.neo4j.rest.graphdb.ExecutingRestRequest.toMap(ExecutingRestRequest.java:169)
    at org.neo4j.rest.graphdb.query.RestCypherQueryEngine.query(RestCypherQueryEngine.java:59)
    at id.co.bippo.mall.admin.RecreatePersonNodesFromLdap.run(RecreatePersonNodesFromLdap.java:90)
    ... 1 more
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@52639bf1; line: 1, column: 2]
    at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at org.neo4j.rest.graphdb.util.JsonHelper.readJson(JsonHelper.java:54)
    ... 5 more

Workaround: Use LIMIT

jexp commented 12 years ago

Interesting, can you have a look at the returned response? What the exception in there is (or in the server logs?)

Thanks a lot for reporting all these.

jexp commented 12 years ago

Might be that the server dies with an OOM, will have streaming enabled in the next version of the Rest-Graph-Database.

systay commented 12 years ago

The issue aside, you really shouldn't do it this way. You should run this query instead:

start n=node:berbatikPerson("*:*")
match n-[r?]->() 
delete n,r 
return distinct id(n)

It will do the same thing, but much faster, thanks to being a single http-command, instead of returning all nodes, and then doing a delete on every one of them.

ceefour commented 12 years ago

today my code LIMIT-ing to 100 records still cause the same error: I want to see the HTTP traffic, how do I set proxy configuration for neo4j-rest-graphdb ?

Exception in thread "main" java.lang.RuntimeException: Error recreating Product nodes
    at id.co.bippo.mall.admin.RecreateProductNodesFromMagento.run(RecreateProductNodesFromMagento.java:199)
    at id.co.bippo.mall.admin.RecreateProductNodesFromMagento.main(RecreateProductNodesFromMagento.java:207)
Caused by: java.lang.RuntimeException: org.codehaus.jackson.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@7786f3b3; line: 1, column: 2]
    at org.neo4j.rest.graphdb.util.JsonHelper.readJson(JsonHelper.java:56)
    at org.neo4j.rest.graphdb.RestAPI.convertRequestResultToEntities(RestAPI.java:230)
    at org.neo4j.rest.graphdb.RestAPI.executeBatch(RestAPI.java:194)
    at id.co.bippo.mall.admin.RecreateProductNodesFromMagento.run(RecreateProductNodesFromMagento.java:92)
    ... 1 more
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@7786f3b3; line: 1, column: 2]
    at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
    at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
    at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
    at org.codehaus.jackson.impl.ReaderBasedParser._handleUnexpectedValue(ReaderBasedParser.java:628)
    at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
    at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2432)
    at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2389)
    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1595)
    at org.neo4j.rest.graphdb.util.JsonHelper.readJson(JsonHelper.java:54)
    ... 4 more

Error log:

Jul 8, 2012 3:51:57 PM org.mortbay.log.Slf4jLog warn
SEVERE: /db/data/batch
org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction
        at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:131)
        at org.neo4j.server.rest.web.BatchOperationService.performBatchOperations(BatchOperationService.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
        at org.neo4j.server.statistic.StatisticFilter.doFilter(StatisticFilter.java:62)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: javax.transaction.HeuristicRollbackException: Failed to commit, transaction rolledback ---> org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node record Node[7364,used=false,rel=11507,prop=9617] still has relationships
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:521)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:406)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:117)
        at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:115)
        ... 37 more
Caused by: org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node record Node[7364,used=false,rel=11507,prop=9617] still has relationships
        at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doPrepare(WriteTransaction.java:175)
        at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.prepare(XaTransaction.java:299)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:435)
        at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
        at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:541)
        at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:444)
        ... 40 more
ceefour commented 12 years ago

Seems like I got another issue, which is a transaction rolled back / cannot delete node error is not propagated properly to neo4j-rest-graphdb client ... will open another bug for this.

ceefour commented 12 years ago

Submitted issue #14 for the "cannot delete node" issue.

As for this one, I'm waiting for how to configure proxy ?

ceefour commented 12 years ago

@systay Thanks for the tip! It's seems 1.8-only syntax, is it ?