apache / incubator-hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
https://hugegraph.apache.org
Apache License 2.0
2.62k stars 517 forks source link

多线程增加6000条边,导致cpu打满 #890

Closed youngkangrui closed 2 years ago

youngkangrui commented 4 years ago

cpu飙高期间的stack信息中,最主要的cpu消耗是下面线程的内容:

"process-callback-handler-21" #3803 prio=5 os_prio=0 tid=0x00007fc62c038000 nid=0x4f022 runnable [0x00007fc40b7f5000]
   java.lang.Thread.State: RUNNABLE
    at org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos$TimeRange.newBuilder(HBaseProtos.java:7493)
    at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTimeRange(ProtobufUtil.java:3264)
    at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:1248)
    at org.apache.hadoop.hbase.shaded.protobuf.RequestConverter.buildGetRequest(RequestConverter.java:180)
    at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:377)
    at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:374)
    at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
    at org.apache.hadoop.hbase.client.HTable.get(HTable.java:384)
    at org.apache.hadoop.hbase.client.HTable.get(HTable.java:358)
    at com.baidu.hugegraph.backend.store.hbase.HbaseSessions$Session.get(HbaseSessions.java:404)
    at com.baidu.hugegraph.backend.store.hbase.HbaseTable.queryById(HbaseTable.java:177)
    at com.baidu.hugegraph.backend.store.hbase.HbaseTable.query(HbaseTable.java:154)
    at com.baidu.hugegraph.backend.store.hbase.HbaseStore.query(HbaseStore.java:197)
    at com.baidu.hugegraph.backend.tx.AbstractTransaction.query(AbstractTransaction.java:104)
    at com.baidu.hugegraph.backend.tx.GraphTransaction.lambda$query$0(GraphTransaction.java:320)
    at com.baidu.hugegraph.backend.tx.GraphTransaction$$Lambda$671/1057907929.apply(Unknown Source)
    at com.baidu.hugegraph.backend.page.QueryList$OptimizedQuery.iterator(QueryList.java:186)
    at com.baidu.hugegraph.backend.page.QueryList.lambda$fetchAll$0(QueryList.java:108)
    at com.baidu.hugegraph.backend.page.QueryList$$Lambda$672/763456237.apply(Unknown Source)
    at com.baidu.hugegraph.iterator.FlatMapperIterator.fetch(FlatMapperIterator.java:55)
    at com.baidu.hugegraph.iterator.WrappedIterator.hasNext(WrappedIterator.java:41)
    at com.baidu.hugegraph.iterator.MapperIterator.fetch(MapperIterator.java:42)
    at com.baidu.hugegraph.iterator.WrappedIterator.hasNext(WrappedIterator.java:41)
    at com.baidu.hugegraph.iterator.FilterIterator.fetch(FilterIterator.java:42)
    at com.baidu.hugegraph.iterator.WrappedIterator.hasNext(WrappedIterator.java:41)
    at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:134)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)
    at org.apache.tinkerpop.gremlin.process.traversal.Traversal.fill(Traversal.java:179)
    **at org.apache.tinkerpop.gremlin.process.traversal.Traversal.toList(Traversal.java:117)**
    at .rns.core.manager.impl.HugeGraphLoader.loadEdge(HugeGraphLoader.java:115)
    at .rns.core.manager.impl.HugeGraphLoader.load(HugeGraphLoader.java:78)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
    - <0x00000000e40a0748> (a java.util.concurrent.ThreadPoolExecutor$Worker)

主要业务功能是先查询节点,在节点之间加入边,对应代码中的是

sourceVertexes=hugeGraph.traversal().V().hasLabel(sourceLabel).has(sourceKey,sourceProperty).toList();
targetVertexes=hugeGraph.traversal().V().hasLabel(targetLabel).has(targetKey,targetProperty).toList()
Vertex sourceVertex = sourceVertexes.get(0);
Vertex targetVertex = targetVertexes.get(0);
sourceVertex.addEdge(edgeLabel,targetVertex);
hugeGraph.tx().commit();
youngkangrui commented 4 years ago

我更换到这段代码后,cpu正常,由于我使用hbase存储,hbase的随机写要快于随机读,所以要将读替换成写

sourceVertex = hugeGraph.addVertex(T.label,sourceLabel,sourceKey, sourceValue);
targetVertex = hugeGraph.addVertex(T.label,targetLabel,targetKey,targetValue);
sourceVertex.addEdge(edgeLabel,targetVertex);
hugeGraph.tx().commit();
javeme commented 4 years ago

@youngkangrui 感谢详细的反馈。是的,防止产生随机读是性能优化的关键点。

另外,如果CPU过高时,可以调小缓存大小降低gc压力,比如:

vertex.cache_capacity=10000
edge.cache_capacity=1000
CPJ-data commented 4 years ago

@youngkangrui 大佬你好 程序之前使用gremlin语法查询单个点耗时比较长

2020-03-12 16:15:37 180082579 [gremlin-server-exec-15] [WARN ] org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine [] - Script compilation g.V().has("eid","e66c7ce822a9ab65f687ab9ef2116b8b") took 12301ms

现在想换成hugeGraph.traversal().V(),看大佬你也是这么操作的,但在操作的时候遇到一点问题不知道怎么搞,就是使用hugegraph-api中hugegraph时,不知道具体参数需要传什么,麻烦大佬帮忙解答一下或者方便的话贴一下大佬你操作hugegraph这块代码参考一下,非常感谢。hugegraph代码如下:

HugeConfig hugeConfig = new HugeConfig(configuration); GraphManager graphManager = new GraphManager(); HugeGraph hugeGraph = graphManager.graph("graph");

youngkangrui commented 4 years ago

@youngkangrui 大佬你好 程序之前使用gremlin语法查询单个点耗时比较长

2020-03-12 16:15:37 180082579 [gremlin-server-exec-15] [WARN ] org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine [] - Script compilation g.V().has("eid","e66c7ce822a9ab65f687ab9ef2116b8b") took 12301ms

现在想换成hugeGraph.traversal().V(),看大佬你也是这么操作的,但在操作的时候遇到一点问题不知道怎么搞,就是使用hugegraph-api中hugegraph时,不知道具体参数需要传什么,麻烦大佬帮忙解答一下或者方便的话贴一下大佬你操作hugegraph这块代码参考一下,非常感谢。hugegraph代码如下:

HugeConfig hugeConfig = new HugeConfig(configuration); GraphManager graphManager = new GraphManager(); HugeGraph hugeGraph = graphManager.graph("graph");

单点查询时间过长的问题没有遇到过,操作hugeGraph的代码如下

    OptionSpace.register("hbase",
            "com.baidu.hugegraph.backend.store.hbase.HbaseOptions");
    SerializerFactory.register("hbase",
            "com.baidu.hugegraph.backend.store.hbase.HbaseSerializer");
    BackendProviderFactory.register("hbase",
            "com.baidu.hugegraph.backend.store.hbase.HbaseStoreProvider");

    PropertiesConfiguration conf = new PropertiesConfiguration();
    conf.setDelimiterParsingDisabled(true);
    conf.setProperty("backend","hbase");
    conf.setProperty("serializer","hbase");
    conf.setProperty("hbase.hosts", "192.169.45.10");
    conf.setProperty("hbase.port","2181");
    conf.setProperty("hbase.znode_parent","/hbase");
    conf.setProperty("store","hugegraph");
    conf.setProperty("store.graph",tableName);
    HugeGraph hugeGraph = HugeFactory.open(conf);
    hugeGraph.initBackend();
github-actions[bot] commented 3 years ago

Due to the lack of activity, the current issue is marked as stale and will be closed after 20 days, any update will remove the stale label