AlbumenJ
commented
1 year ago 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的https://github.com/apache/dubbo/issues/6294
这个的完整报错堆栈可以贴一下吗
Closed HTFX-Cloud closed 10 months ago
AlbumenJ
commented
1 year ago 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的https://github.com/apache/dubbo/issues/6294
这个的完整报错堆栈可以贴一下吗
HTFX-Cloud
commented
1 year ago 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的#6294
这个的完整报错堆栈可以贴一下吗
从10点开始出现Warn级别日志
[Dubbo-thread-2] ~ WARN org.apache.dubbo.remoting.exchange.support.DefaultFuture 177 received - [DUBBO] The timeout response finally returned at 2022-10-19 10:01:09.345, response status is 20, channel: /10.177.56.133:43182 -> /10.177.56.133:20882, please check provider side for detailed result., dubbo version: 2.7.8, current host: 161.2.56.133
以及error的完整信息
10-19 10:27:32,632 [pool-7-thread-305] ~db4ed7dab49b42dcaa809c8f212a2e5e DEBUG a.b.c.bd.consumer.PrepareForTaskExecutor 110 prepareExecutionParameter - DataOps: key=projectId,value=null
10-19 10:27:43,391 [pool-7-thread-4442] ~416ce361b8e9466f894a46a6e34adace ERROR a.b.c.bd.manager.TaskExecutionService 199 executeTask - 1次调用后非文件存储异常信息 org.apache.dubbo.rpc.RpcException: Failed to invoke the method executeTask in the service a.b.c.bd.api.IEngineService. Tried 1 times of the providers [10.177.56.133:20882] (1/4) from the registry 10.177.56.130:9908 on the consumer 161.2.56.133 using the dubbo version 2.7.8. Last error is: Fail to invoke remote method: executeTask, provider: dubbo://10.177.56.133:20882/engineService?anyhost=true&application=dbd-app&category=providers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=public&init=true&interface=a.b.c.bd.api.IEngineService&metadata-type=remote&methods=reRunTask,executeTask&path=engineService&payload=104857600&pid=2378&protocol=dubbo&qos.enable=false®ister.ip=161.2.56.133&release=2.7.8&remote.application=dbd-app&retries=0&revision=5.3-MASTER-20220608.095739-141&side=consumer&sticky=false&timeout=30000×tamp=1655987866520&version=5.0, cause: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:260) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0.executeTask(proxy0.java) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0$$FastClassBySpringCGLIB$$c197671e.invoke(
10-19 10:27:32,627 [pool-7-thread-1405] ~720d2d06a8744456b4d740d5cd1be846 INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 410 doBegin - NODE[720d2d06a8744456b4d740d5cd1be846] 执行任务开始步骤：初始化任务变量！ 10-19 10:27:32,627 [pool-7-thread-4967] ~357eeb0785e2443a8d0f3594c8307ad5 DEBUG a.b.c.bd.dao.impl.TaskOperationDao 1503 getAllStatus - getAllStatus:[ ] 10-19 10:27:32,617 [pool-7-thread-2957] ~4031a4934f6c47cab380c66e2e46f4df INFO a.b.c.bd.logic.impl.TodoTaskOperationService 269 processTodoTask - 处理代办时，workflowKey=03a7a64a0ce14b9b95d264036d964d89,workflowCurrentKey=69A7FA817268477B9C0DBF3686396C4F,nodeKey=fee5cfb39e9748de9fa152f55bc12b18,taskRunId=d1555c00314c4a759d333316327cecac对应的任务启动时的项目信息为{projectId=null, spaceId=0, WfbloodRerunInfo=} 10-19 10:27:32,600 [pool-7-thread-1700] ~c885fe8c32434010a0eb9f06892c1b6f INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 675 doReleaseRes - NODE[c885fe8c32434010a0eb9f06892c1b6f] 执行任务释放资源步骤：释放资源！
AlbumenJ
commented
1 year ago 从日志上看是 provider 到 nacos 的心跳断开了，导致 nacos 推送该 provider 下线，但是有在途请求然后报错
可以检查下为什么 nacos 注册中心会推送下线信息
HTFX-Cloud
commented
1 year ago 从日志上看是 provider 到 nacos 的心跳断开了，导致 nacos 推送该 provider 下线，但是有在途请求然后报错
可以检查下为什么 nacos 注册中心会推送下线信息
我看了一下nacos集群（130、131、132）三个节点上10.19日的naming.log文件。其中在10:27:04.914和10:27:20.508两个时间三个节点都存在
2022-10-19 10:27:04.914 INFO [com.alibaba.nacos.client.naming.updater:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IEngineService:5.0:public -> [{"clusterName":"DEFAULT","enabled":true,"ephemeral":true,"healthy":true,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000,"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IEngineService:5.0:public","ip":"10.177.56.133","ipDeleteTimeout":30000,"metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"com.bonc.dataplatform.buw.dispatcher.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866520"},"port":20882,"serviceName":"DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IEngineService:5.0:public","weight":1.0}]
和
2022-10-19 10:27:20.508 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService:5.0:public -> [{"clusterName":"DEFAULT","enabled":true,"ephemeral":true,"healthy":true,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000,"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService:5.0:public","ip":"10.177.56.133","ipDeleteTimeout":30000,"metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,startWorkflow,startScheduleWorkflow,executeTaskForTemp,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866761"},"port":20882,"serviceName":"DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService:5.0:public","weight":1.0}]摘除133节点（也就是上文提到负载调用不平衡的A节点）
并且在nacos集群中的132节点存在如下的日志
2022-10-19 10:27:05.066 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] received push data: {"type":"dom","data":"{\"hosts\":[{\"ip\":\"10.177.56.132\",\"port\":20882,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"53008\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987760581\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.134\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"31262\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987989629\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.135\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"56132\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655988109440\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true}],\"dom\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"name\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"cacheMillis\":10000,\"lastRefTime\":1666146425065,\"checksum\":\"9190a7c1dcdb5fce2375f5282bd116ae\",\"useSpecifiedURL\":false,\"clusters\":\"\",\"env\":\"\",\"metadata\":{}}","lastRefTime":56135268660146131} from /10.177.56.132
2022-10-19 10:27:05.066 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] received push data: {"type":"dom","data":"{\"hosts\":[{\"ip\":\"10.177.56.132\",\"port\":20882,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"53008\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987760581\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.134\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"31262\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987989629\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.135\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"56132\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655988109440\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true}],\"dom\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"name\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"cacheMillis\":10000,\"lastRefTime\":1666146425067,\"checksum\":\"9190a7c1dcdb5fce2375f5282bd116ae\",\"useSpecifiedURL\":false,\"clusters\":\"\",\"env\":\"\",\"metadata\":{}}","lastRefTime":56135268660146131} from /10.177.56.132
2022-10-19 10:27:05.085 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.133","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866520"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:05.085 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.133","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866520"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:05.086 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] current ips:(3) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.132","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"53008","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987760581"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.134","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"31262","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987989629"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.135","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"56132","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655988109440"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:05.529 WARN [com.alibaba.nacos.naming.push.receiver:client.naming] write DEFAULT_GROUP%40%40providers%3Aa.b.c.bd.api.IEngineService%3A5.0%3Apublic conflict;retry time: 1
2022-10-19 10:27:05.529 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] current ips:(3) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.132","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"53008","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987760581"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.134","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"31262","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987989629"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.135","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"56132","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655988109440"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:20.858 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] received push data: {"type":"dom","data":"{\"hosts\":[{\"ip\":\"10.177.56.132\",\"port\":20882,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"53008\",\"interface\":\"a.b.c.bd.api.IWorkflowWebService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"WorkflowWebService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987760900\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.134\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"31262\",\"interface\":\"a.b.c.bd.api.IWorkflowWebService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"WorkflowWebService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987989873\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.135\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"56132\",\"interface\":\"a.b.c.bd.api.IWorkflowWebService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"WorkflowWebService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655988109818\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"ephemeral\":true}],\"dom\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"name\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"cacheMillis\":10000,\"lastRefTime\":1666146440499,\"checksum\":\"aef2359632324193f6bbced5589e6846\",\"useSpecifiedURL\":false,\"clusters\":\"\",\"env\":\"\",\"metadata\":{}}","lastRefTime":56135284087143858} from /10.177.56.132
2022-10-19 10:27:20.859 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public -> [{"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.133","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,startWorkflow,startScheduleWorkflow,executeTaskForTemp,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866761"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:20.860 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] current ips:(3) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public -> [{"instanceId":"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.132","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"53008","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987760900"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.134","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"31262","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987989873"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.135","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"56132","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655988109818"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 11:36:36.582 WARN [com.alibaba.nacos.client.naming.updater:client.naming] failed to request http://10.177.56.130:9908/nacos/v1/ns/instance/list?app=unknown&healthyOnly=false&namespaceId=10_177_37_101_hh_process_cjzh_src_realtime&clientIP=10.177.56.132&serviceName=DEFAULT_GROUP%40%40providers%3Acom.bonc.dataplatform.buw.executor.api.ITaskService%3A5.0%3AshellScript&udpPort=51084&encoding=UTF-8 from 10.177.56.130
2022-10-19 11:36:36.583 WARN [com.alibaba.nacos.client.naming.updater:client.naming] failed to request http://10.177.56.130:9908/nacos/v1/ns/instance/list?app=unknown&healthyOnly=false&namespaceId=10_177_37_111_hh_process_cjzh_src_realtime&clientIP=10.177.56.132&serviceName=DEFAULT_GROUP%40%40providers%3Acom.bonc.dataplatform.buw.executor.api.ITaskService%3A5.0%3AshellScript&udpPort=48579&encoding=UTF-8 from 10.177.56.130
2022-10-19 11:36:36.583 ERROR [com.alibaba.nacos.client.naming.updater:client.naming] [NA] failed to request
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_181]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_181]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_181]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.
AlbumenJ
commented
1 year ago 可以到 https://github.com/alibaba/nacos/issues 提个 issue 看下是什么问题，这个日志看起来是 nacos 内部同步用的
Environment
Steps to reproduce this issue