search

apache / dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
https://dubbo.apache.org/
Apache License 2.0
40.52k stars 26.43k forks source link

Version or Group mismatch on retry when an error occurs #9439

Closed chenzhiguo closed 2 years ago

chenzhiguo commented 2 years ago

Environment

Steps to reproduce this issue

  1. Define 2 the same name service: group2 / com.xxxx.demo.DemoService: 2.0.0: 20880 and group1 / com.xxxx.demo.DemoService: 1.0.0: 20880.
  2. Configure consumer consumer service: com.xxxx.demo.DemoService, group: , version: , configure the number of retry = 3.
  3. Invoke DemoService one method and throws runtime exception to enter the retry logic.

Pls. provide [GitHub address] to reproduce this issue.

Expected Behavior

Actual Behavior

If there is an exception, please attach the exception trace:

org.apache.dubbo.remoting.RemotingException: Not found exported service: group1/com.xxxx.demo.DemoService:2.0.0:20880 in [group2/com.xxxx.demo.HelloService:2.0.0:20880, group2/com.xxxx.demo.DemoService:2.0.0:20880, group1/com.xxxx.demo.DemoService:1.0.0:20880], may be version or group mismatch , channel: consumer: /10.0.192.45:49030 --> provider: /10.0.128.32:20880, message:RpcInvocation [methodName=getException, parameterTypes=[], arguments=[], attachments={input=815, path=com.xxxx.demo.DemoService, retries=1, remote.application=dubbo-consumer-demo, service.name=com.xxxx.demo.DemoService, namespace=jiashuo-1214, dubbo=2.0.2, interface=com.xxxx.demo.DemoService, System-TagMap={"system-label-ip":"10.0.192.45","system-label-mesh":"mesh-group","system-label-deployment":"consumer-sub","system-label-group":"appGroup1","system-label-namespace":"jiashuo-1214","system-label-method":"getException","system-label-zone":"zone1","system-label-app":"dubbo-consumer-app-jiashuo","system-label-instance":"dubbo-consumer-sub-58d6f68d96-4k29h","system-label-service":"com.xxxx.demo.DemoService","system-label-cluster":"cluster-jiashuo-sub"}, version=2.0.0, timeout=2000, group=group1}]
    at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:267)
    at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:123)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:100)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:175)
    at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
    at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
    at java.base/java.lang.Thread.run(Thread.java:829)
] with root cause

org.apache.dubbo.remoting.RemotingException: org.apache.dubbo.remoting.RemotingException: Not found exported service: group1/com.xxxx.demo.DemoService:2.0.0:20880 in [group2/com.xxxx.demo.HelloService:2.0.0:20880, group2/com.xxxx.demo.DemoService:2.0.0:20880, group1/com.xxxx.demo.DemoService:1.0.0:20880], may be version or group mismatch , channel: consumer: /10.0.192.45:49030 --> provider: /10.0.128.32:20880, message:RpcInvocation [methodName=getException, parameterTypes=[], arguments=[], attachments={input=815, path=com.xxxx.demo.DemoService, retries=1, remote.application=dubbo-consumer-demo, service.name=com.xxxx.demo.DemoService, namespace=jiashuo-1214, dubbo=2.0.2, interface=com.xxxx.demo.DemoService, System-TagMap={"system-label-ip":"10.0.192.45","system-label-mesh":"mesh-group","system-label-deployment":"consumer-sub","system-label-group":"appGroup1","system-label-namespace":"jiashuo-1214","system-label-method":"getException","system-label-zone":"zone1","system-label-app":"dubbo-consumer-app-jiashuo","system-label-instance":"dubbo-consumer-sub-58d6f68d96-4k29h","system-label-service":"com.xxxx.demo.DemoService","system-label-cluster":"cluster-jiashuo-sub"}, version=2.0.0, timeout=2000, group=group1}]
org.apache.dubbo.remoting.RemotingException: Not found exported service: group1/com.xxxx.demo.DemoService:2.0.0:20880 in [group2/com.xxxx.demo.HelloService:2.0.0:20880, group2/com.xxxx.demo.DemoService:2.0.0:20880, group1/com.xxxx.demo.DemoService:1.0.0:20880], may be version or group mismatch , channel: consumer: /10.0.192.45:49030 --> provider: /10.0.128.32:20880, message:RpcInvocation [methodName=getException, parameterTypes=[], arguments=[], attachments={input=815, path=com.xxxx.demo.DemoService, retries=1, remote.application=dubbo-consumer-demo, service.name=com.xxxx.demo.DemoService, namespace=jiashuo-1214, dubbo=2.0.2, interface=com.xxxx.demo.DemoService, System-TagMap={"system-label-ip":"10.0.192.45","system-label-mesh":"mesh-group","system-label-deployment":"consumer-sub","system-label-group":"appGroup1","system-label-namespace":"jiashuo-1214","system-label-method":"getException","system-label-zone":"zone1","system-label-app":"dubbo-consumer-app-jiashuo","system-label-instance":"dubbo-consumer-sub-58d6f68d96-4k29h","system-label-service":"com.xxxx.demo.DemoService","system-label-cluster":"cluster-jiashuo-sub"}, version=2.0.0, timeout=2000, group=group1}]
    at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:267)
    at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:123)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:100)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:175)
    at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
    at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
    at java.base/java.lang.Thread.run(Thread.java:829)

    at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:207)
    at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:170)
    at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:158)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleResponse(HeaderExchangeHandler.java:60)
    at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:181)
    at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
    at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
    at org.apache.dubbo.common.threadpool.ThreadlessExecutor$RunnableWrapper.run(ThreadlessExecutor.java:196)
    at org.apache.dubbo.common.threadpool.ThreadlessExecutor.waitAndDrain(ThreadlessExecutor.java:99)
    at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:179)
    at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:61)
    at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
    at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:91)
    at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
    at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
    at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
    at com.xxxx.meshware.dubbo2.filter.LocalAddressFilter.invoke(LocalAddressFilter.java:24)
    at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
    at com.xxxx.meshware.dubbo2.filter.ConsumerTraceFilter.invoke(ConsumerTraceFilter.java:41)
    at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
    at com.xxxx.meshware.dubbo2.filter.JmsfConsumerFilter.invoke(JmsfConsumerFilter.java:111)
    at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
    at org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69)
    at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:61)
    at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
    at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:79)
    at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:265)
    at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47)
    at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92)
    at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:93)
    at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:170)
    at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:96)
    at org.apache.dubbo.common.bytecode.proxy0.getException(proxy0.java)
    at com.xxxx.dubbo.consumer.controller.ConsumerController.exception(ConsumerController.java:43)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:829)
chenzhiguo commented 2 years ago

I think there is a code in the doInvoke method of DubboInvoker:

inv.setAttachment(VERSION_KEY, version);

About this position, it missing set group.

AlbumenJ commented 2 years ago

Retry only work in one group. Group selection only failback when the providers size is 0.

chenzhiguo commented 2 years ago

Retry only work in one group. Group selection only failback when the providers size is 0.

You mean this is normal abnormal and normal error address? I don't think so. This error sometimes appears, when matches the wrong version and group, but when matches the correct version and group, it is not. I mean, the version and group in Invocation should not only modify one. Because only one is changed, it is likely to have no such instance.

AlbumenJ commented 2 years ago

Multi group is design like multi registry subscription, work only for fallback if address is empty. If you need retry for all providers, specify them as the same group.

CrazyHZM commented 2 years ago

The above answer should solve your problem, temporarily close the issue, if there is still a problem, you can reopen it.