大数据量(超过10000)执行g.V().hasLabel('XXX').drop()删除报错

tmljob commented 3 years ago

Expected behavior 期望表现

在hubble执行gremlin任务，删除某个顶点的数据，能够正常删除。

g.V().hasLabel('rel_D_wuqianw').drop()

说明： 1、顶点数据量5kw； 2、server0.11.2+hubble1.5.0

Actual behavior 实际表现

任务执行失败，报如下错误：com.baidu.hugegraph.exception.LimitExceedException: Vertices size has reached tx capacity 10000

server的log如下：

 2020-12-15 16:50:37 19858643 [task-worker-2] [WARN ] com.baidu.hugegraph.task.HugeTask [] - An exception occurred when running task: 18
7657 com.baidu.hugegraph.exception.LimitExceedException: Vertices size has reached tx capacity 10000
7658         at com.baidu.hugegraph.backend.tx.GraphTransaction.checkTxVerticesCapacity(GraphTransaction.java:1780) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7659         at com.baidu.hugegraph.backend.tx.GraphTransaction.beforeWrite(GraphTransaction.java:252) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7660         at com.baidu.hugegraph.backend.tx.GraphTransaction.removeVertex(GraphTransaction.java:630) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7661         at com.baidu.hugegraph.StandardHugeGraph.removeVertex(StandardHugeGraph.java:471) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7662         at com.baidu.hugegraph.structure.HugeVertex.remove(HugeVertex.java:406) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7663         at org.apache.tinkerpop.gremlin.process.traversal.step.filter.DropStep.filter(DropStep.java:68) ~[gremlin-core-3.4.3.jar:3.4.3]
7664         at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:38) ~[gremlin-core-3.4.3.jar:3.4.3]
7665         at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143) ~[gremlin-core-3.4.3.jar:3.4.3]
7666         at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:197) ~[gremlin-core-3.4.3.jar:3.4.3]
7667         at com.baidu.hugegraph.job.GremlinJob.execute(GremlinJob.java:82) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7668         at com.baidu.hugegraph.job.UserJob.call(UserJob.java:29) ~[hugegraph-core-0.11.2.jar:0.11.2.0]
7669         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
7670         at com.baidu.hugegraph.task.HugeTask.run(HugeTask.java:281) [hugegraph-core-0.11.2.jar:0.11.2.0]
7671         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
7672         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
7673         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
7674         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
7675         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
~

cmdares commented 3 years ago

目前，我们的场景是有一批数据写入时候弄错了，需要在2亿中删除2000万个边。当时没搞定，一批一批删除也不现实。最后把其他类型的数据重新灌入。关注该问题，也希望官方能给个最佳实践方案。

javeme commented 3 years ago

大量删除数据一般会导致性能下降（产生大量墓碑），建议使用TTL或者标记删除。

实在需要删除的话，需要将drop改为迭代删除：

t = g.V().hasLabel('rel_D_wuqianw')
while (t.hasNext()) {
  t.next().remove()
  graph.tx().commitIfGtSize(500)
}

tmljob commented 3 years ago

大量删除数据一般会导致性能下降（产生大量墓碑），建议使用TTL或者标记删除。

实在需要删除的话，需要将drop改为迭代删除：
t = g.V().hasLabel('rel_D_wuqianw')
while (t.hasNext()) {
  t.next().remove()
  graph.tx().commitIfGtSize(500)
}

这个迭代删除，在Hubble以异步任务执行时，为什么会自动加上limit(250)的限制？ @javeme

{
    "task_name": "t = g.V().hasLabel('rel_D_wuqianw').limit(250)",
    "task_progress": 0,
    "task_create": 1608083914414,
    "task_status": "success",
    "task_update": 1608083914763,
    "task_result": "[]",
    "task_retries": 0,
    "id": 26,
    "task_type": "gremlin",
    "task_callable": "com.baidu.hugegraph.job.GremlinJob",
    "task_input": "{\"gremlin\":\"t = g.V().hasLabel('rel_D_wuqianw').limit(250)\\nwhile (t.hasNext()) {\\n  t.next().remove()\\n  graph.tx().commitIfGtSize(500)\\n}\",\"aliases\":{\"daas\":\"graph\"},\"bindings\":{},\"language\":\"gremlin-groovy\"}",
    "task_server": "server-1"
}

Linary commented 3 years ago

@lxb1111 关注一下执行任务的时候是否会加上.limit(250)

lxb1111 commented 3 years ago

好的

lxb1111 commented 3 years ago

已经确认，确实存在相关问题，谢谢反馈。我们会尽快修复完成！

javeme commented 2 years ago

在Hubble以异步任务执行时，为什么会自动加上limit(250)的限制？

limit问题在 https://github.com/hugegraph/hugegraph-hubble/pull/285 中修复了。

javeme commented 2 years ago

大量删除顶点建议使用异步任务，并且使用迭代方式删除，参考：https://github.com/hugegraph/hugegraph/issues/1298#issuecomment-745170866

apache / incubator-hugegraph

大数据量(超过10000)执行g.V().hasLabel('XXX').drop()删除报错 #1298

Expected behavior 期望表现

Actual behavior 实际表现