search

alibaba / spring-cloud-alibaba

Spring Cloud Alibaba provides a one-stop solution for application development for the distributed solutions of Alibaba middleware.
https://sca.aliyun.com
Apache License 2.0
27.83k stars 8.32k forks source link

error with using dubbo.consumer.check=false configuration #800

Closed linghaijun closed 4 years ago

linghaijun commented 5 years ago

my consumer config: dubbo: protocols: dubbo: name: dubbo port: 28800 registry: address: spring-cloud://localhost consumer: timeout: 60000 check: false

Which Component Nacos Discovery

org.springframework.cloud spring-cloud-starter-dubbo 0.9.0.RELEASE 
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-alibaba-nacos-discovery</artifactId>
        <version>0.9.0.RELEASE</version>
    </dependency>

Describe the bug I don't want to check the provider depended when starting the consumer. So I add dubbo.consumer.check=false to my bootstrap.yml. And it worked when starting my consumer .But when my consumer call the api service of provider.I got the exception:

org.apache.dubbo.rpc.RpcException: Failed to invoke the method add in the service com.kris.service.api.UserService. No provider available for the service com.kris.service.api.UserService from registry localhost:9090 on the consumer 10.80.29.127 using the dubbo version 2.7.1. Please check if the providers have been started and registered. at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.checkInvokers(AbstractClusterInvoker.java:265) ~[dubbo-2.7.1.jar:2.7.1] at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:57) ~[dubbo-2.7.1.jar:2.7.1] at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:242) ~[dubbo-2.7.1.jar:2.7.1] at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:76) ~[dubbo-2.7.1.jar:2.7.1] at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:57) ~[dubbo-2.7.1.jar:2.7.1] at org.apache.dubbo.common.bytecode.proxy0.add(proxy0.java) ~[dubbo-2.7.1.jar:2.7.1] at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:45005) ~[na:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162] at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor$ReferenceBeanInvocationHandler.invoke(ReferenceAnnotationBeanPostProcessor.java:165) ~[dubbo-2.7.1.jar:2.7.1] at com.sun.proxy.$Proxy109.add(Unknown Source) ~[na:na] at com.kris.controller.UserController.test(UserController.java:16) ~[classes/:na] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:45005) ~[na:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90) ~[spring-boot-actuator-2.1.3.RELEASE.jar:2.1.3.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:117) ~[spring-boot-actuator-2.1.3.RELEASE.jar:2.1.3.RELEASE] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106) ~[spring-boot-actuator-2.1.3.RELEASE.jar:2.1.3.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) ~[tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.StandardContextValve.__invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:41002) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) [tomcat-embed-core-9.0.16.jar:9.0.16] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.16.jar:9.0.16] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.16.jar:9.0.16] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]

It seems that my consumer can't find the provider.

But it can work without the config dubbo.consumer.check=false

fangjian0423 commented 5 years ago

No provider available for the service ...

You must startup your provider application.

linghaijun commented 5 years ago

I have started my provider application already.

fangjian0423 commented 5 years ago

you mean startup provider after consumer, but the consumer can't find the provider? refer https://github.com/alibaba/spring-cloud-alibaba/pull/813, we are trying to fix it.

linghaijun commented 5 years ago

Yes, thank you, I will pay attention to it.

linghaijun commented 4 years ago

I think it is the cause of the problem. As nacos retains the offline service ,when running a consumer first, initializedServices has already inited the offline service. And when the provider start up, consumer's listener is notified and initializedServices.contains(serviceName) is true. So it won't do the initialization of the code below:

initSubscribedDubboMetadataService(serviceName);
initDubboRestServiceMetadataRepository(serviceName);

So,I think the below function should be optimized.

DubboServiceMetadataRepository.initializeMetadata()

/**
     * Initialize the metadata of Dubbo Services
     */
    public void initializeMetadata(String serviceName) {
        synchronized (monitor) {
            if (initializedServices.contains(serviceName)) {
                if (logger.isDebugEnabled()) {
                    logger.debug(
                            "The metadata of Dubbo service[name : {}] has been initialized",
                            serviceName);
                }
            }
            else {
                if (logger.isInfoEnabled()) {
                    logger.info(
                            "The metadata of Dubbo service[name : {}] is about to be initialized",
                            serviceName);
                }

                // Keep the order in following invocations
                initSubscribedDubboMetadataService(serviceName);
                initDubboRestServiceMetadataRepository(serviceName);
                // mark this service name having been initialized
                initializedServices.add(serviceName);
            }
        }
    }
linghaijun commented 4 years ago

Add two temp solution.

1.Copy a class of DubboServiceMetadataRepository to overwrite the origin one(The package should be the same to the origin one) to the project. and modify the functioninitializeMetadata(String serviceName) to this:

public void initializeMetadata(String serviceName) {
        synchronized (monitor) {
            if (logger.isInfoEnabled()) {
                logger.info(
                        "The metadata of Dubbo service[name : {}] is about to be initialized",
                        serviceName);
            }

            // Keep the order in following invocations
            initSubscribedDubboMetadataService(serviceName);
            initDubboRestServiceMetadataRepository(serviceName);
            // mark this service name having been initialized
            initializedServices.add(serviceName);
        }
    }

2.Delete the offline services on Nacos before fully starting the all the services.

tyq0010 commented 4 years ago

you mean startup provider after consumer, but the consumer can't find the provider? refer #813, we are trying to fix it.

When I introduced Spring Cloud Alibaba to our company's project (with more than 10 microservices), I found that this problem still arose.(consumer tip "No Provider...", but actually the corresponding provider has been started).Every time this happens, I have to manually restart the consumer. （Sometimes the heartbeat task can detect and reconnect automatically, but sometimes it doesn't.）

Version: 2.1.1.BUILD-SNAPSHOT + (cherry pick #973). I can guarantee that the final jar package contains the fix code, such as DubboGenericServiceFactory -> public synchronized void destroy(String serviceName)

I have been busy recently, so I don't have time to describe in detail how to reproduce this problem. But in general, start consumers and producers at the same time. And it is easier to appear when change the provider's port number each time, and finding ways to make startup times longer, such as exposing more services or subscribing to more services.

linghaijun commented 4 years ago

you mean startup provider after consumer, but the consumer can't find the provider? refer #813, we are trying to fix it.

When I introduced Spring Cloud Alibaba to our company's project (with more than 10 microservices), I found that this problem still arose.(consumer tip "No Provider...", but actually the corresponding provider has been started).Every time this happens, I have to manually restart the consumer. （Sometimes the heartbeat task can detect and reconnect automatically, but sometimes it doesn't.）

Version: 2.1.1.BUILD-SNAPSHOT + (cherry pick #973). I can guarantee that the final jar package contains the fix code, such as DubboGenericServiceFactory -> public synchronized void destroy(String serviceName)

I have been busy recently, so I don't have time to describe in detail how to reproduce this problem. But in general, start consumers and producers at the same time. And it is easier to appear when change the provider's port number each time, and finding ways to make startup times longer, such as exposing more services or subscribing to more services.

Do you setdubbo.consumer.check=false ?
If you set it true. I think you even can't startup your consumer when you start the provider and consumer at the same time. If you set it false. I think you have the same trouble. The microservies must start in order. For example if A service rely B service，you should start up B fisrt. If you see B is onlone at nacos, then you start A.

The simple two temp solution is what I wrote yesterday.

tyq0010 commented 4 years ago

@linghaijun Yes, I've set dubbo.consumer.check=false, refer #904. And I tried your solution 1. but it doesn't work for me. The problem seems not so simple.

tyq0010 commented 4 years ago

In fact, fangjian0423 fixed it. And I tested it, and it seemed to work. But I don't know why not now.

I don't know which version you're using. To ensure that your code contains the #973 fix, you can add this line (This method is added in #973)DubboGenericServiceFactory.class.getMethod("destroy", String.class); before Springapplication.run(...);, if no NoSuchMethodException is thrown, your code contains the fix. But it now appears that even with the fix code, the problem remains.

linghaijun commented 4 years ago

@tyq0010 I has not try 2.1.1.BUILD-SNAPSHOT. The version is 2.1.0.RELEASE I think you can try to debug DubboServiceMetadataRepository.initializeMetadata(String serviceName) when you start the provider after starting up the consumer. And see the result of initializedServices.contains(serviceName) , if true, consumer won't init the provider's metainfo. So the consumer can't find the provider when request.

Translate: 我没有去尝试2.1.1.BUILD-SNAPSHOT版本，我用的是2.1.0.RELEASE 下面是我调试观察到的： 你可以先看一下这个方法。 DubboServiceMetadataRepository.initializeMetadata(String serviceName) 用中文简单的说一下，就是nacos里面一直保留了不在线的服务，你的consumer启动的时候会去初始化provider 的一些服务的元数据，这个时候provider不在线，但是还是会把未上线的provider写到initializedServices这里，等到你的provider启动的时候，consumer监听到，会再次init，但是这时候判断initializedServices.contains(serviceName)为true，所以不再初始化provider的服务元数据。导致了你的consumer找不到服务的提供者。 所以你可以首先试一下，你把隐藏空服务关掉，然后把nacos里面的不在线的服务先删光，然后你先启动consumer，再起provider。看看应用启动后consumer能不能正常调用provider。如果可以调用，说明你遇到的问题跟我一样。

tyq0010 commented 4 years ago

@linghaijun 我的情况可能和你的不太一样，我用的是2.1.1.BUILD-SNAPSHOT版本(greenwich分支)，我的版本是包含了修复代码的(之前我是cherry pick #973这个提交，而在10月30日那天，官方从master分支同步了大量代码到geenwich分支，其中就包含了修复代码)。

根据我今晚反复测试的结果，只有在spring.cloud.nacos.discovery.group设置了分组(consumer和provider的group设置成一样)的情况下，才会出现找不到服务者的问题。 否则，无论consumer和provider启动顺序怎样，甚至同时启动，服务都是正常的。

之前我都是在自己电脑上测试，没有出现问题。 而当用于公司项目的时候， 因为需要设置分组，才发现有这个问题。 并且dubbo.provider.group, dubbo.consumer.group, dubbo.config-center.group, dubbo.metadata-report.group这些都可以设置分组，不用关心启动顺序。 而只要spring.cloud.nacos.discovery.group设置了分组，就只能先启动provider再启动consumer，目前我这边的情况就是这样。

linghaijun commented 4 years ago

@fangjian0423 I think this bug has been fixed when processing common invoke in 2.1.1，but still appearing when my dubbo client processes ‘generic invoke‘

fangjian0423 commented 4 years ago

If the problem is not resolved, please create a new issue.