apache / dubbo-samples

samples for Apache Dubbo
Apache License 2.0
2.23k stars 1.91k forks source link

集成测试VersionServiceStarIT有时会失败 #378

Closed zrlw closed 3 years ago

zrlw commented 3 years ago

github构建集成测试断言失败:

2021-10-05T02:50:59.8126068Z java.lang.AssertionError
2021-10-05T02:50:59.8126612Z    at org.junit.Assert.fail(Assert.java:86)
2021-10-05T02:50:59.8127249Z    at org.junit.Assert.assertTrue(Assert.java:41)
2021-10-05T02:50:59.8127939Z    at org.junit.Assert.assertTrue(Assert.java:52)
2021-10-05T02:50:59.8129233Z    at org.apache.dubbo.samples.version.VersionServiceStarIT.test(VersionServiceStarIT.java:48)

一个接口有两个不同版本的dubbo服务,负载均衡为默认的随机模式,客户端连续调用10次服务,测试断言每个服务均被调用过,但断言失败了。 从日志看ZK有告警(不一定和上述断言失败有关系):

2021-10-05T02:50:59.9117330Z [05/10/21 02:50:43:043 UTC] Curator-Framework-0  WARN utils.ZKPaths: The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.

网上搜了一下,说是curator和zk版本不兼容导致的问题。 从github日志看,zk客户端使用的是3.4.13,curator framework版本是4.0.1,zk服务端版本是3.7.0:

2021-10-05T02:50:59.7529466Z [05/10/21 02:50:52:052 UTC] main  INFO utils.Compatibility: Running in ZooKeeper 3.4.x compatibility mode
2021-10-05T02:50:59.7530246Z [05/10/21 02:50:52:052 UTC] main  INFO imps.CuratorFrameworkImpl: Starting
2021-10-05T02:50:59.7531703Z [05/10/21 02:50:52:052 UTC] main  INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT  
<== zk client版本3.4.13
2021-10-05T02:50:59.7533172Z [05/10/21 02:50:52:052 UTC] main  INFO zookeeper.ZooKeeper: Client environment:host.name=dubbo-samples-version-consumer
2021-10-05T02:50:59.7534769Z [05/10/21 02:50:52:052 UTC] main  INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_302
2021-10-05T02:50:59.7535628Z [05/10/21 02:50:52:052 UTC] main  INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2021-10-05T02:50:59.7536790Z [05/10/21 02:50:52:052 UTC] main  INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/openjdk-8/jre
2021-10-05T02:50:59.7551508Z [05/10/21 02:50:52:052 UTC] main  INFO zookeeper.ZooKeeper: Client environment:java.class.path=dubbo-test-runner.jar:/usr/local/dubbo/app/test-classes:/usr/local/dubbo/app/classes:/usr/local/dubbo/app/dependency/jline-0.9.94.jar:/usr/local/dubbo/app/dependency/jackson-core-asl-1.9.13.jar:/usr/local/dubbo/app/dependency/slf4j-api-1.7.25.jar:/usr/local/dubbo/app/dependency/fastjson-1.2.70.jar:/usr/local/dubbo/app/dependency/curator-client-4.0.1.jar:/usr/local/dubbo/app/dependency/commons-logging-1.2.jar:/usr/local/dubbo/app/dependency/spring-test-4.3.30.RELEASE.jar:/usr/local/dubbo/app/dependency/slf4j-log4j12-1.7.25.jar:/usr/local/dubbo/app/dependency/spring-context-4.3.30.RELEASE.jar:/usr/local/dubbo/app/dependency/junit-4.12.jar:/usr/local/dubbo/app/dependency/curator-x-discovery-4.0.1.jar:/usr/local/dubbo/app/dependency/spring-expression-4.3.30.RELEASE.jar:/usr/local/dubbo/app/dependency/gson-2.8.5.jar:/usr/local/dubbo/app/dependency/dubbo-2.7.14-SNAPSHOT.jar:/usr/local/dubbo/app/dependency/spring-core-4.3.30.RELEASE.jar:/usr/local/dubbo/app/dependency/spring-aop-4.3.30.RELEASE.jar:/usr/local/dubbo/app/dependency/hamcrest-core-1.3.jar:/usr/local/dubbo/app/dependency/spring-beans-4.3.30.RELEASE.jar:/usr/local/dubbo/app/dependency/netty-all-4.1.66.Final.jar:/usr/local/dubbo/app/dependency/zookeeper-3.4.13.jar:/usr/local/dubbo/app/dependency/audience-annotations-0.5.0.jar:/usr/local/dubbo/app/dependency/snakeyaml-1.20.jar:/usr/local/dubbo/app/dependency/log4j-1.2.16.jar:/usr/local/dubbo/app/dependency/curator-recipes-4.0.1.jar:/usr/local/dubbo/app/dependency/javassist-3.23.1-GA.jar:/usr/local/dubbo/app/dependency/spring-context-support-1.0.11.jar:/usr/local/dubbo/app/dependency/curator-framework-4.0.1.jar:/usr/local/dubbo/app/dependency/guava-20.0.jar:/usr/local/dubbo/app/dependency/jackson-mapper-asl-1.9.13.jar  
<== curator-framework版本4.0.1
... ... ...
2021-10-05T02:51:00.0674913Z 2021-10-05 02:50:37,206 [myid:1] - INFO  [main:Environment@98] - Server environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC  
<== zk服务端版本3.7.0
2021-10-05T02:51:00.0676208Z 2021-10-05 02:50:37,206 [myid:1] - INFO  [main:Environment@98] - Server environment:host.name=zookeeper
2021-10-05T02:51:00.0677198Z 2021-10-05 02:50:37,207 [myid:1] - INFO  [main:Environment@98] - Server environment:java.version=11.0.12
2021-10-05T02:51:00.0678229Z 2021-10-05 02:50:37,207 [myid:1] - INFO  [main:Environment@98] - Server environment:java.vendor=Oracle Corporation
zrlw commented 3 years ago

要兼容zk 3.4.x,需要使用curator 4.2.0 (目前maven上4.2.x的curator只有4.2.0版本)

ZooKeeper Version 3.4.x Compatibility https://curator.apache.org/zk-compatibility-34.html

ZooKeeper 3.4.x is now at end-of-life. Consequently, the latest versions of Curator have removed support for it. If you wish to use Curator with ZooKeeper 3.4.x you should pin to version 4.2.x of Curator. Curator 4.2.x supports ZooKeeper 3.4.x ensembles in a soft-compatibility mode. To use this mode you must exclude ZooKeeper when adding Curator to your dependency management tool.

Maven

<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-recipes</artifactId>
    <version>4.2.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
    </exclusions>
</dependency>
zrlw commented 3 years ago

如果升级集成测试使用的curator和zk版本,可以参考一下dubbo-config-spring的pom.xml文件配置的单元测试使用的curator和zk版本:

        <curator5_version>5.1.0</curator5_version>
        <zookeeper_version>3.6.0</zookeeper_version>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-framework</artifactId>
            <version>${curator5_version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>${curator5_version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-x-discovery</artifactId>
            <version>${curator5_version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>${zookeeper_version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-test</artifactId>
            <version>${curator5_version}</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.zookeeper</groupId>
                    <artifactId>zookeeper</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
zrlw commented 3 years ago

github日志显示version的3个集成测试类先后执行顺序是VersionService2IT、VersionServiceStarIT、VersionService1IT,只有VersionServiceStarIT失败了,从日志看VersionServiceStarIT从注册中心拿到了2个服务实例:

[05/10/21 02:50:55:055 UTC] main  INFO zookeeper.ZookeeperRegistry:  [DUBBO] Notify urls for subscribe url consumer://192.168.80.5/org.apache.dubbo.samples.version.api.VersionService?application=version-consumer-1&category=providers,configurators,routers&dubbo=2.0.2&init=false&interface=org.apache.dubbo.samples.version.api.VersionService&metadata-type=remote&methods=sayHello&pid=19&release=2.7.14-SNAPSHOT&revision=*&side=consumer&sticky=false&timestamp=1633402255073&version=*, urls: [dubbo://192.168.80.3:20880/org.apache.dubbo.samples.version.api.VersionService?anyhost=true&application=version-provider&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=org.apache.dubbo.samples.version.api.VersionService&metadata-type=remote&methods=sayHello&pid=20&release=2.7.14-SNAPSHOT&revision=1.0.0&service.name=ServiceBean:/org.apache.dubbo.samples.version.api.VersionService:1.0.0&side=provider&timestamp=1633402243059&token=11c5796d-1931-4414-a827-a3fda9bbbe25&version=1.0.0, dubbo://192.168.80.4:20882/org.apache.dubbo.samples.version.api.VersionService?anyhost=true&application=version-provider2&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=org.apache.dubbo.samples.version.api.VersionService&metadata-type=remote&methods=sayHello&pid=20&release=2.7.14-SNAPSHOT&revision=2.0.0&service.name=ServiceBean:/org.apache.dubbo.samples.version.api.VersionService:2.0.0&side=provider&timestamp=1633402243665&token=6f21afe9-7e85-4703-a635-c8ccadc30755&version=2.0.0, empty://192.168.80.5/org.apache.dubbo.samples.version.api.VersionService?application=version-consumer-1&category=configurators&dubbo=2.0.2&init=false&interface=org.apache.dubbo.samples.version.api.VersionService&metadata-type=remote&methods=sayHello&pid=19&release=2.7.14-SNAPSHOT&revision=*&side=consumer&sticky=false&timestamp=1633402255073&version=*, empty://192.168.80.5/org.apache.dubbo.samples.version.api.VersionService?application=version-consumer-1&category=routers&dubbo=2.0.2&init=false&interface=org.apache.dubbo.samples.version.api.VersionService&metadata-type=remote&methods=sayHello&pid=19&release=2.7.14-SNAPSHOT&revision=*&side=consumer&sticky=false&timestamp=1633402255073&version=*], dubbo version: 2.7.14-SNAPSHOT, current host: 192.168.80.5

其中192.168.80.4:20882是VersionService2IT使用2.0.0版本的接口,连接在执行VersionService2IT测试时已经建好,192.168.80.3:20880是1.0.0版本的接口,日志显示需要新建连接:

[05/10/21 02:50:55:055 UTC] main  INFO transport.AbstractClient:  [DUBBO] Start NettyClient /192.168.80.5 connect to the server /192.168.80.3:20880, dubbo version: 2.7.14-SNAPSHOT, current host: 192.168.80.5
<== 创建连接到1.0.0版本服务
[05/10/21 02:50:55:055 UTC] main  INFO config.ReferenceConfig:  [DUBBO] Refer dubbo service org.apache.dubbo.samples.version.api.VersionService from url dubbo://192.168.80.5/org.apache.dubbo.samples.version.api.VersionService?application=version-consumer-1&dubbo=2.0.2&init=false&interface=org.apache.dubbo.samples.version.api.VersionService&metadata-type=remote&methods=sayHello&pid=19&register.ip=192.168.80.5&release=2.7.14-SNAPSHOT&revision=*&side=consumer&sticky=false&timestamp=1633402255073&version=*, dubbo version: 2.7.14-SNAPSHOT, current host: 192.168.80.5
<== master分支ReferenceConfig的createProxy方法
[05/10/21 02:50:55:055 UTC] DubboSaveMetadataReport-thread-1  INFO zookeeper.ZookeeperMetadataReport:  [DUBBO] store consumer metadata. Identifier : org.apache.dubbo.metadata.report.identifier.MetadataIdentifier@7fd8366a; definition: {init=false, side=consumer, metadata-type=remote, application=version-consumer-1, release=2.7.14-SNAPSHOT, methods=sayHello, sticky=false, dubbo=2.0.2, interface=org.apache.dubbo.samples.version.api.VersionService, version=*, revision=*}, dubbo version: 2.7.14-SNAPSHOT, current host: 192.168.80.5
[05/10/21 02:50:55:055 UTC] main  INFO support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker: Bean 'versionService' of type [org.apache.dubbo.common.bytecode.proxy0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[05/10/21 02:50:55:055 UTC] main  INFO support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker: Bean 'org.apache.dubbo.samples.version.VersionServiceStarIT' of type [org.apache.dubbo.samples.version.VersionServiceStarIT] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[05/10/21 02:50:55:055 UTC] NettyClientWorker-4-2  INFO netty4.NettyClientHandler:  [DUBBO] The connection of /192.168.80.5:44230 -> /192.168.80.3:20880 is established., dubbo version: 2.7.14-SNAPSHOT, current host: 192.168.80.5
<== 已连接1.0.0版本的服务
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.725 s <<< FAILURE! - in org.apache.dubbo.samples.version.VersionServiceStarIT

从日志顺序看,可能是1.0.0版本的连接尚未完成建立就开始测试了,所以只调用了版本2.0.0的服务。