search

apache / incubator-hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
https://hugegraph.apache.org
Apache License 2.0
2.62k stars 516 forks source link

[Bug] Raft 模式部署后删除边 Label 失败 #1739

Open fcser opened 2 years ago

fcser commented 2 years ago

Bug Type (问题类型)

gremlin (结果不合预期)

Before submit

Environment (环境信息)

3个节点，配置了raft协议 raft.mode=true raft.safe_read=true raft.use_snapshot=true raft.endpoint=10...27:8281 raft.group_peers=10...27:8281,10...235:8281,10...40:8281

Expected & Actual behavior (期望与实际表现)

image com.baidu.hugegraph.HugeException: Failed to remove edges

Vertex/Edge example (问题点 / 边数据举例)

No response

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

No response

imbajin commented 2 years ago

删除 500w 条边是么? 你可以看看后端 server 的日志里有什么相关信息么

javeme commented 2 years ago

@fcser 感谢反馈，可以提供一下复现流程吗？

fcser commented 2 years ago

删除 500w 条边是么? 你可以看看后端 server 的日志里有什么相关信息么

2022-01-26 16:12:44 [task-worker-2] [WARN] c.b.h.b.s.r.RaftBackendStore - Failed to execute query '`Query * from EDGE page 'HosxOjBEWW9zb0JTZ3CCCAEAizE6QXc2eW5COHQzQwAAAAAAAAH0', limit 500`': Status[ETIMEDOUT<1010>: read-index request timeout]
com.baidu.hugegraph.backend.BackendException: Failed to do raft read-index: Status[ETIMEDOUT<1010>: read-index request timeout]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:225) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.alipay.sofa.jraft.closure.ReadIndexClosure$TimeoutTask.run(ReadIndexClosure.java:153) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:609) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:677) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[jraft-core-1.3.9.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2022-01-26 16:12:44 [task-worker-2] [ERROR] c.b.h.b.Transaction - Failed to remove edges
com.baidu.hugegraph.backend.BackendException: Failed to execute query: `Query * from EDGE page 'HosxOjBEWW9zb0JTZ3CCCAEAizE6QXc2eW5COHQzQwAAAAAAAAH0', limit 500`
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:237) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:209) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.query(RaftBackendStore.java:143) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.AbstractTransaction.query(AbstractTransaction.java:163) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.query(GraphTransaction.java:533) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.queryEdgesFromBackend(GraphTransaction.java:981) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.cache.CachedGraphTransaction.queryEdgesFromBackend(CachedGraphTransaction.java:279) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.queryEdges(GraphTransaction.java:944) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.traverseByLabel(GraphTransaction.java:2037) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.traverseEdgesByLabel(GraphTransaction.java:1990) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.removeEdges(GraphTransaction.java:1968) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.job.schema.EdgeLabelRemoveJob.removeEdgeLabel(EdgeLabelRemoveJob.java:71) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.job.schema.EdgeLabelRemoveJob.execute(EdgeLabelRemoveJob.java:42) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.job.SysJob.call(SysJob.java:29) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201]
        at com.baidu.hugegraph.task.HugeTask.run(HugeTask.java:287) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_201]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: com.baidu.hugegraph.backend.BackendException: Failed to do raft read-index: Status[ETIMEDOUT<1010>: read-index request timeout]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:225) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.alipay.sofa.jraft.closure.ReadIndexClosure$TimeoutTask.run(ReadIndexClosure.java:153) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:609) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:677) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[jraft-core-1.3.9.jar:?]
        ... 1 more
2022-01-26 16:12:44 [task-worker-2] [WARN] c.b.h.t.HugeTask - An exception occurred when running task: 2
com.baidu.hugegraph.HugeException: Failed to remove edges
        at com.baidu.hugegraph.backend.tx.GraphTransaction.removeEdges(GraphTransaction.java:1976) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.job.schema.EdgeLabelRemoveJob.removeEdgeLabel(EdgeLabelRemoveJob.java:71) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.job.schema.EdgeLabelRemoveJob.execute(EdgeLabelRemoveJob.java:42) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.job.SysJob.call(SysJob.java:29) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201]
        at com.baidu.hugegraph.task.HugeTask.run(HugeTask.java:287) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_201]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: com.baidu.hugegraph.backend.BackendException: Failed to execute query: `Query * from EDGE page 'HosxOjBEWW9zb0JTZ3CCCAEAizE6QXc2eW5COHQzQwAAAAAAAAH0', limit 500`
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:237) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:209) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.query(RaftBackendStore.java:143) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.AbstractTransaction.query(AbstractTransaction.java:163) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.query(GraphTransaction.java:533) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.queryEdgesFromBackend(GraphTransaction.java:981) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.cache.CachedGraphTransaction.queryEdgesFromBackend(CachedGraphTransaction.java:279) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.queryEdges(GraphTransaction.java:944) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.traverseByLabel(GraphTransaction.java:2037) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.traverseEdgesByLabel(GraphTransaction.java:1990) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.baidu.hugegraph.backend.tx.GraphTransaction.removeEdges(GraphTransaction.java:1968) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        ... 10 more
Caused by: com.baidu.hugegraph.backend.BackendException: Failed to do raft read-index: Status[ETIMEDOUT<1010>: read-index request timeout]
        at com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:225) ~[hugegraph-core-0.12.0.jar:0.12.0.0]
        at com.alipay.sofa.jraft.closure.ReadIndexClosure$TimeoutTask.run(ReadIndexClosure.java:153) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:609) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:677) ~[jraft-core-1.3.9.jar:?]
        at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[jraft-core-1.3.9.jar:?]
        ... 1 more
2022-01-26 16:12:44 [JRaft-FSMCaller-Disruptor-0] [WARN] c.a.s.j.c.ReadIndexClosure - A timeout read-index response finally returned: Status[OK].
2022-01-26 16:21:52 [task-
javeme commented 2 years ago

@fcser 贡献反馈，其它raft节点的日志是否也可以提供一下？

isliudong commented 2 months ago

同款异常，但是我是停止后无法启动

isliudong commented 2 months ago

2024-06-21 14:14:24 [main] [WARN] c.b.h.b.s.r.RaftNode - Waiting for raft group 'default' election cost 3.091s 2024-06-21 14:14:24 [main] [INFO] c.b.h.b.s.r.RaftBackendStoreProvider - The raft store is started 2024-06-21 14:14:26 [main] [WARN] c.b.h.b.s.r.RaftBackendStore - Failed to execute query 'Query * from SECONDARY_INDEX where id prefix with 0x2d333a7e6261636b656e645f696e666f00': Status[ETIMEDOUT<1010>: read-index request timeout] com.baidu.hugegraph.backend.BackendException: Failed to do raft read-index: Status[ETIMEDOUT<1010>: read-index request timeout] at com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:225) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.alipay.sofa.jraft.closure.ReadIndexClosure$TimeoutTask.run(ReadIndexClosure.java:153) ~[jraft-core-1.3.9.jar:?] at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:609) ~[jraft-core-1.3.9.jar:?] at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:677) ~[jraft-core-1.3.9.jar:?] at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[jraft-core-1.3.9.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] 2024-06-21 14:14:26 [main] [INFO] o.a.t.g.s.GremlinServer - Shutting down OpProcessor[] 2024-06-21 14:14:26 [main] [INFO] o.a.t.g.s.GremlinServer - Shutting down OpProcessor[session] 2024-06-21 14:14:26 [main] [ERROR] c.b.h.d.HugeGraphServer - HugeRestServer start error: com.baidu.hugegraph.backend.BackendException: Failed to execute query: Query * from SECONDARY_INDEX where id prefix with 0x2d333a7e6261636b656e645f696e666f00 at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:237) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.queryByRaft(RaftBackendStore.java:209) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.store.raft.RaftBackendStore.query(RaftBackendStore.java:143) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.tx.AbstractTransaction.query(AbstractTransaction.java:163) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.tx.SchemaIndexTransaction.queryByName(SchemaIndexTransaction.java:99) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.tx.SchemaIndexTransaction.query(SchemaIndexTransaction.java:77) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.tx.SchemaTransaction.getSchema(SchemaTransaction.java:391) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.cache.CachedSchemaTransaction.getSchema(CachedSchemaTransaction.java:256) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.tx.SchemaTransaction.getPropertyKey(SchemaTransaction.java:136) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.store.BackendStoreSystemInfo.info(BackendStoreSystemInfo.java:72) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.backend.store.BackendStoreSystemInfo.exists(BackendStoreSystemInfo.java:92) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.baidu.hugegraph.core.GraphManager.checkBackendVersionOrExit(GraphManager.java:354) ~[hugegraph-api-0.12.0.jar:0.67.0.0] at com.baidu.hugegraph.core.GraphManager.(GraphManager.java:102) ~[hugegraph-api-0.12.0.jar:0.67.0.0] at com.baidu.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:112) ~[hugegraph-api-0.12.0.jar:0.67.0.0] at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:74) ~[jersey-server-2.25.1.jar:?] at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:81) ~[jersey-server-2.25.1.jar:?] at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:1180) ~[jersey-server-2.25.1.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:357) ~[jersey-container-grizzly2-http-2.25.1.jar:?] at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:398) ~[grizzly-http-server-2.4.4.jar:2.4.4] at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:293) ~[grizzly-http-server-2.4.4.jar:2.4.4] at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:269) ~[grizzly-http-server-2.4.4.jar:2.4.4] at com.baidu.hugegraph.server.RestServer.start(RestServer.java:68) ~[hugegraph-api-0.12.0.jar:0.67.0.0] at com.baidu.hugegraph.server.RestServer.start(RestServer.java:175) ~[hugegraph-api-0.12.0.jar:0.67.0.0] at com.baidu.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:34) ~[hugegraph-dist-0.12.0.jar:?] at com.baidu.hugegraph.dist.HugeGraphServer.(HugeGraphServer.java:71) ~[hugegraph-dist-0.12.0.jar:?] at com.baidu.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:119) ~[hugegraph-dist-0.12.0.jar:?] Caused by: com.baidu.hugegraph.backend.BackendException: Failed to do raft read-index: Status[ETIMEDOUT<1010>: read-index request timeout] at com.baidu.hugegraph.backend.store.raft.RaftBackendStore$1.run(RaftBackendStore.java:225) ~[hugegraph-core-0.12.0.jar:0.12.0.0] at com.alipay.sofa.jraft.closure.ReadIndexClosure$TimeoutTask.run(ReadIndexClosure.java:153) ~[jraft-core-1.3.9.jar:?] at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:609) ~[jraft-core-1.3.9.jar:?] at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:677) ~[jraft-core-1.3.9.jar:?] at com.alipay.sofa.jraft.util.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395) ~[jraft-core-1.3.9.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] 2024-06-21 14:14:26 [main] [INFO] o.a.t.g.s.GremlinServer - Shutting down OpProcessor[traversal] 2024-06-21 14:14:26 [main] [INFO] o.a.t.g.s.GremlinServer - Shutting down thread pools.