OpenSPG / openspg

OpenSPG is a Knowledge Graph Engine developed by Ant Group in collaboration with OpenKG, based on the SPG (Semantic-enhanced Programmable Graph) framework. Core Capabilities: 1) domain model constrained knowledge modeling, 2) facts and logic fused representation, 3) KAG will be natively supported soon, so please stay tuned...
https://spg.openkg.cn/en-US
Apache License 2.0
657 stars 76 forks source link

Knext问题 #106

Closed GitYohoo closed 8 months ago

GitYohoo commented 8 months ago

我无法在案例目录中使用knex指令,是因为我的knext版本问题吗 (openspg_venv) root@DESKTOP-RPSG79T:/code/python/knext/knext/examples/riskmining# knext Traceback (most recent call last): File "/openspg_venv/bin/knext", line 5, in from knext.command.knext_cli import _main File "/openspg_venv/lib/python3.8/site-packages/knext/init.py", line 17, in init_env() File "/openspg_venv/lib/python3.8/site-packages/knext/common/env.py", line 59, in init_env load_operator() File "/openspg_venv/lib/python3.8/site-packages/knext/common/env.py", line 134, in load_operator register_from_package(builder_operator_path, BaseOp) File "/openspg_venv/lib/python3.8/site-packages/knext/common/class_register.py", line 52, in register_from_package _register(path, root, files, class_type) File "/openspg_venv/lib/python3.8/site-packages/knext/common/class_register.py", line 29, in _register module = importlib.import_module(module_name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/code/python/knext/knext/examples/riskmining/builder/operator/cert_link_operator.py", line 15, in from knext.builder.operator.op import LinkOp ModuleNotFoundError: No module named 'knext.builder'

baifuyu commented 8 months ago

我无法在案例目录中使用knex指令,是因为我的knext版本问题吗 (openspg_venv) root@DESKTOP-RPSG79T:/code/python/knext/knext/examples/riskmining# knext Traceback (most recent call last): File "/openspg_venv/bin/knext", line 5, in from knext.command.knext_cli import _main File "/openspg_venv/lib/python3.8/site-packages/knext/init.py", line 17, in init_env() File "/openspg_venv/lib/python3.8/site-packages/knext/common/env.py", line 59, in init_env load_operator() File "/openspg_venv/lib/python3.8/site-packages/knext/common/env.py", line 134, in load_operator register_from_package(builder_operator_path, BaseOp) File "/openspg_venv/lib/python3.8/site-packages/knext/common/class_register.py", line 52, in register_from_package _register(path, root, files, class_type) File "/openspg_venv/lib/python3.8/site-packages/knext/common/class_register.py", line 29, in _register module = importlib.import_module(module_name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/code/python/knext/knext/examples/riskmining/builder/operator/cert_link_operator.py", line 15, in from knext.builder.operator.op import LinkOp ModuleNotFoundError: No module named 'knext.builder'

抱歉,github上的最新master代码,knext的包结构有些调整,但是这个docker还是老的knext镜像版本造成的。

可以先在本地的openspg代码库中执行下 git checkout v0.0.2-beta2,再尝试下执行上面的命令。我们尽快发布下最新的knext docker 镜像版本

GitYohoo commented 8 months ago

我无法在案例目录中使用knex指令,是因为我的knext版本问题吗 (openspg_venv) root@DESKTOP-RPSG79T:/code/python/knext/knext/examples/riskmining# knext Traceback (most recent call last): File "/openspg_venv/bin/knext", line 5, in from knext.command.knext_cli import _main File "/openspg_venv/lib/python3.8/site-packages/knext/init.py", line 17, in init_env() File "/openspg_venv/lib/python3.8/site-packages/knext/common/env.py", line 59, in init_env load_operator() File "/openspg_venv/lib/python3.8/site-packages/knext/common/env.py", line 134, in load_operator register_from_package(builder_operator_path, BaseOp) File "/openspg_venv/lib/python3.8/site-packages/knext/common/class_register.py", line 52, in register_from_package _register(path, root, files, class_type) File "/openspg_venv/lib/python3.8/site-packages/knext/common/class_register.py", line 29, in _register module = importlib.import_module(module_name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/code/python/knext/knext/examples/riskmining/builder/operator/cert_link_operator.py", line 15, in from knext.builder.operator.op import LinkOp ModuleNotFoundError: No module named 'knext.builder'

抱歉,github上的最新master代码,knext的包结构有些调整,但是这个docker还是老的knext镜像版本造成的。

可以先在本地的openspg代码库中执行下 git checkout v0.0.2-beta2,再尝试下执行上面的命令。我们尽快发布下最新的knext docker 镜像版本

我按照你的提示确实运行了 但是出现新的问题: 2024-01-26 06:26:42,713 [] [] [main] INFO c.a.o.c.i.g.GraphStoreClientDriverManager - registerDriver: com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClientDriver@55b0dcab 2024-01-26 06:26:42,715 [] [] [main] INFO c.a.o.c.u.DriverManagerUtils - DriverManager.initialize: cloudext.graphstore.drivers = null 2024-01-26 06:26:42,715 [] [] [main] INFO c.a.o.c.i.g.GraphStoreClientDriverManager - graph-store DriverManager initialized 2024-01-26 06:26:43,077 [] [] [main] ERROR c.b.b.c.BrpcProxy - exception : com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.BrpcProxy.intercept(BrpcProxy.java:268) at com.antgroup.tugraph.TuGraphDbService$$EnhancerByCGLIB$$3f8070da.HandleRequest() at com.antgroup.tugraph.TuGraphDbRpcClient$TuGraphSingleRpcClient.(TuGraphDbRpcClient.java:419) at com.antgroup.tugraph.TuGraphDbRpcClient.(TuGraphDbRpcClient.java:63) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClient.initTuGraphClient(TuGraphStoreClient.java:139) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClient.(TuGraphStoreClient.java:84) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClientDriver.innerConnect(TuGraphStoreClientDriver.java:36) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClientDriver.innerConnect(TuGraphStoreClientDriver.java:22) at com.antgroup.openspg.common.util.cloudext.CachedCloudExtClientDriver.connect(CachedCloudExtClientDriver.java:29) at com.antgroup.openspg.cloudext.interfaces.graphstore.GraphStoreClientDriverManager.getClient(GraphStoreClientDriverManager.java:49) at com.antgroup.openspg.builder.runner.local.physical.sink.impl.GraphStoreSinkWriter.doInit(GraphStoreSinkWriter.java:59) at com.antgroup.openspg.builder.core.physical.BasePhysicalNode.init(BasePhysicalNode.java:71) at com.antgroup.openspg.builder.runner.local.LocalBuilderRunner.init(LocalBuilderRunner.java:76) at com.antgroup.openspg.builder.runner.local.LocalBuilderMain.run(LocalBuilderMain.java:148) at com.antgroup.openspg.builder.runner.local.LocalBuilderMain.main(LocalBuilderMain.java:69) Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.RpcFuture.get(RpcFuture.java:178) at com.baidu.brpc.interceptor.LoadBalanceInterceptor.rpcCore(LoadBalanceInterceptor.java:93) at com.baidu.brpc.interceptor.LoadBalanceInterceptor.invokeRpc(LoadBalanceInterceptor.java:82) at com.baidu.brpc.interceptor.LoadBalanceInterceptor.aroundProcess(LoadBalanceInterceptor.java:62) at com.baidu.brpc.interceptor.DefaultInterceptorChain.intercept(DefaultInterceptorChain.java:43) at com.baidu.brpc.interceptor.AbstractInterceptor.aroundProcess(AbstractInterceptor.java:35) at com.baidu.brpc.interceptor.ClientTraceInterceptor.aroundProcess(ClientTraceInterceptor.java:40) at com.baidu.brpc.interceptor.DefaultInterceptorChain.intercept(DefaultInterceptorChain.java:43) at com.baidu.brpc.client.BrpcProxy.intercept(BrpcProxy.java:266) ... 14 common frames omitted Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.handler.RpcClientHandler$2.run(RpcClientHandler.java:109) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.handler.RpcClientHandler.channelRead0(RpcClientHandler.java:68) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ... 2 common frames omitted Caused by: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.protocol.standard.BaiduRpcProtocol.decode(BaiduRpcProtocol.java:190) at com.baidu.brpc.protocol.standard.BaiduRpcProtocol.decode(BaiduRpcProtocol.java:70) at com.baidu.brpc.client.handler.RpcClientHandler.channelRead0(RpcClientHandler.java:56) ... 17 common frames omitted 2024-01-26 06:26:43,077 [] [] [main] ERROR c.a.o.b.r.l.LocalBuilderMain - unknown exception. java.lang.RuntimeException: init TuGraph Client failed at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClient.initTuGraphClient(TuGraphStoreClient.java:142) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClient.(TuGraphStoreClient.java:84) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClientDriver.innerConnect(TuGraphStoreClientDriver.java:36) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClientDriver.innerConnect(TuGraphStoreClientDriver.java:22) at com.antgroup.openspg.common.util.cloudext.CachedCloudExtClientDriver.connect(CachedCloudExtClientDriver.java:29) at com.antgroup.openspg.cloudext.interfaces.graphstore.GraphStoreClientDriverManager.getClient(GraphStoreClientDriverManager.java:49) at com.antgroup.openspg.builder.runner.local.physical.sink.impl.GraphStoreSinkWriter.doInit(GraphStoreSinkWriter.java:59) at com.antgroup.openspg.builder.core.physical.BasePhysicalNode.init(BasePhysicalNode.java:71) at com.antgroup.openspg.builder.runner.local.LocalBuilderRunner.init(LocalBuilderRunner.java:76) at com.antgroup.openspg.builder.runner.local.LocalBuilderMain.run(LocalBuilderMain.java:148) at com.antgroup.openspg.builder.runner.local.LocalBuilderMain.main(LocalBuilderMain.java:69) Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.BrpcProxy.intercept(BrpcProxy.java:277) at com.antgroup.tugraph.TuGraphDbService$$EnhancerByCGLIB$$3f8070da.HandleRequest() at com.antgroup.tugraph.TuGraphDbRpcClient$TuGraphSingleRpcClient.(TuGraphDbRpcClient.java:419) at com.antgroup.tugraph.TuGraphDbRpcClient.(TuGraphDbRpcClient.java:63) at com.antgroup.openspg.cloudext.impl.graphstore.tugraph.TuGraphStoreClient.initTuGraphClient(TuGraphStoreClient.java:139) ... 10 common frames omitted Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.RpcFuture.get(RpcFuture.java:178) at com.baidu.brpc.interceptor.LoadBalanceInterceptor.rpcCore(LoadBalanceInterceptor.java:93) at com.baidu.brpc.interceptor.LoadBalanceInterceptor.invokeRpc(LoadBalanceInterceptor.java:82) at com.baidu.brpc.interceptor.LoadBalanceInterceptor.aroundProcess(LoadBalanceInterceptor.java:62) at com.baidu.brpc.interceptor.DefaultInterceptorChain.intercept(DefaultInterceptorChain.java:43) at com.baidu.brpc.interceptor.AbstractInterceptor.aroundProcess(AbstractInterceptor.java:35) at com.baidu.brpc.interceptor.ClientTraceInterceptor.aroundProcess(ClientTraceInterceptor.java:40) at com.baidu.brpc.interceptor.DefaultInterceptorChain.intercept(DefaultInterceptorChain.java:43) at com.baidu.brpc.client.BrpcProxy.intercept(BrpcProxy.java:266) ... 14 common frames omitted Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.handler.RpcClientHandler$2.run(RpcClientHandler.java:109) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) Caused by: com.baidu.brpc.exceptions.RpcException: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.client.handler.RpcClientHandler.channelRead0(RpcClientHandler.java:68) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ... 2 common frames omitted Caused by: com.baidu.brpc.exceptions.BadSchemaException: not valid magic head for brpc at com.baidu.brpc.protocol.standard.BaiduRpcProtocol.decode(BaiduRpcProtocol.java:190) at com.baidu.brpc.protocol.standard.BaiduRpcProtocol.decode(BaiduRpcProtocol.java:70) at com.baidu.brpc.client.handler.RpcClientHandler.channelRead0(RpcClientHandler.java:56) ... 17 common frames omitted 我发现docker内部的时间与实际时间不一致,相差8小时,是因为时区导致的。我尝试修改了所有5个容器内部的时间,使用date查询已经没有问题。但是knext中时间依然不对。如下: Reason: HTTP response headers: HTTPHeaderDict({'Remote': '172.18.0.2', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 26 Jan 2024 07:49:50 GMT', 'Connection': 'close'}) HTTP response body: "project name:RiskMining has already exist, please change the project name and try again "

(openspg_venv) root@DESKTOP-RPSG79T:/code/python/knext/knext/examples/riskmining# date Fri Jan 26 15:49:52 CST 2024

**我不确定前面提到的问题是不是因为时间导致的,但这也算一个bug

前面的问题该如何解决呢?**

baifuyu commented 8 months ago

我发现docker内部的时间与实际时间不一致,相差8小时,是因为时区导致的。我尝试修改了所有5个容器内部的时间,使用date查询已经没有问题。但是knext中时间依然不对。如下: Reason: HTTP response headers: HTTPHeaderDict({'Remote': '172.18.0.2', 'Content-Type': 'application/json;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 26 Jan 2024 07:49:50 GMT', 'Connection': 'close'}) HTTP response body: "project name:RiskMining has already exist, please change the project name and try again "

可以再执行下 docker images 看下这2个镜像是否最新的,目前这个报错应该是tugraph的版本和client没匹配上造成的。如果可以这个问题,可以删除镜像重新拉取下 image

还有个openspg/openspg-python也得看下是否最新的,尽量使用最新的版本。

GitYohoo commented 8 months ago

我已经将镜像更新到了最新版本,并且尝试了删除重新拉取所有5个images,以及重启容器等等 image 我也尝试了不同分支的代码,比如master、v0.002-beta5、v0.002-beta2等等。但是还是会出现同样的问题 image

baifuyu commented 8 months ago

我已经将镜像更新到了最新版本,并且尝试了删除重新拉取所有5个images,以及重启容器等等 image 我也尝试了不同分支的代码,比如master、v0.002-beta5、v0.002-beta2等等。但是还是会出现同样的问题 image

仔细看了下几个image确实是最新的。我们电脑复现不出这个问题。下面的日志是在执行knext builder execute 时报的错吗?以及帮忙再提供下os相关的信息。顺便再帮忙执行下docker ps -a,看下docker进程是否正常

GitYohoo commented 8 months ago

docker ps -a 我确实是在执行knext builder execute时报错的 另外我是在wsl2中运行的,下面是详细信息 image image docker信息如下: image

会不会确实是wsl2的原因?

baifuyu commented 8 months ago

docker ps -a 我确实是在执行knext builder execute时报错的 另外我是在wsl2中运行的,下面是详细信息 image image docker信息如下: image

会不会确实是wsl2的原因?

tugraph我们在官网里默认设置了端口是7070和9090,如下: image

但是上图显示了您这边用了9091端口,导致knext端去链接9090时报错 image

有两种方式可以修改正确:

  1. 将该端口改为9090,并且注意server端也得需要改为9090,如下: image
  2. 还是坚持使用9091,则需要将server端也需要改为9091,同时在knext端执行以下命令告知knext使用9090端口:knext config set --global graph_store_url="tugraph://11.166.207.228:9091?graphName=default&timeout=50000&accessId=admin&accessKey=73@TuGraph"
baifuyu commented 8 months ago

@GitYohoo 方便理解,以下是目前openspg单机版的部署架构图

image

GitYohoo commented 8 months ago
  1. 还是坚持使用9091,则需要将server端也需要改为9091,同时在knext端执行以下命令告知knext使用9090端口:knext config set --global graph_store_url="tugraph://11.166.207.228:9091?graphName=default&timeout=50000&accessId=admin&accessKey=73@TuGraph" 可以了 非常感谢