spring-cloud / spring-cloud-sleuth

Distributed tracing for spring cloud
https://spring.io/projects/spring-cloud-sleuth
Apache License 2.0
1.78k stars 782 forks source link

sleuthStreamSpanReporter sometimes not get post processed by MessagingAnnotationPostProcessor #719

Closed gdong42 closed 6 years ago

gdong42 commented 7 years ago

Hi team,

I encountered a wired situation that my project is not able to send spans over Spring Cloud Stream to Kafka. After investigation for a while, I found that it was due to bean sleuthStreamSpanReporter somehow was not getting post-processed by MessagingAnnotationPostProcessor, which in turn causes following method not getting polled at all:

    @InboundChannelAdapter(value = SleuthSource.OUTPUT, poller = @Poller(POLLER))
    public Spans poll() {
...

thus the spans have no chance to be collected and send to kafka via stream. The other project using same dependencies runs fine. I noticed that the problematic one initializes sleuthStreamSpanReporter bean before MessagingAnnotationPostProcessor is getting initialized. It also prints following info:

2017-09-30 21:45:48.011  INFO [example-api,,,] 7618 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'sleuthStreamSpanReporter' of type [org.springframework.cloud.sleuth.stream.StreamSpanReporter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

I guess this might be relevant. However, I could not locate the root cause why this happens. Can you please help me and give some insights.

Thank you.


This is my dependencies:

[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ example-api ---
[INFO] com.company.example:example-api:jar:1.0.0-SNAPSHOT
[INFO] +- com.company.example:example-proto:jar:1.0.0-SNAPSHOT:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.6.1:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.4.0:compile
[INFO] |  |  +- com.google.guava:guava:jar:20.0:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java-util:jar:3.4.0:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
[INFO] |  |  \- io.grpc:grpc-protobuf-lite:jar:1.6.1:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.6.1:compile
[INFO] |  \- com.company.common:commons-error-handling:jar:0.1.0-SNAPSHOT:compile
[INFO] |     \- io.netty:netty-common:jar:4.1.14.Final:compile
[INFO] +- org.jolokia:jolokia-core:jar:1.3.7:compile
[INFO] |  \- com.googlecode.json-simple:json-simple:jar:1.1.1:compile
[INFO] +- org.springframework.boot:spring-boot-starter-actuator:jar:1.5.6.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:1.5.6.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:1.5.6.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:1.5.6.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:1.5.6.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.1.11:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.1.11:compile
[INFO] |  |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.25:compile
[INFO] |  |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  |  \- org.slf4j:log4j-over-slf4j:jar:1.7.25:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.17:runtime
[INFO] |  \- org.springframework.boot:spring-boot-actuator:jar:1.5.6.RELEASE:compile
[INFO] |     \- org.springframework:spring-context:jar:4.3.10.RELEASE:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-config:jar:1.3.2.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-starter:jar:1.2.3.RELEASE:compile
[INFO] |  |  +- org.springframework.cloud:spring-cloud-context:jar:1.2.3.RELEASE:compile
[INFO] |  |  \- org.springframework.security:spring-security-rsa:jar:1.0.3.RELEASE:compile
[INFO] |  |     \- org.bouncycastle:bcpkix-jdk15on:jar:1.55:compile
[INFO] |  |        \- org.bouncycastle:bcprov-jdk15on:jar:1.55:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-config-client:jar:1.3.2.RELEASE:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.0:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.8.9:compile
[INFO] |     \- com.fasterxml.jackson.core:jackson-core:jar:2.8.9:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:1.5.6.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:1.5.6.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.16:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.16:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.16:compile
[INFO] |  +- org.hibernate:hibernate-validator:jar:5.3.5.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.1.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.3:compile
[INFO] |  +- org.springframework:spring-web:jar:4.3.10.RELEASE:compile
[INFO] |  |  +- org.springframework:spring-aop:jar:4.3.10.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-beans:jar:4.3.10.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:4.3.10.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:4.3.10.RELEASE:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-consul-discovery:jar:1.2.1.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-starter-consul:jar:1.2.1.RELEASE:compile
[INFO] |  |  +- org.springframework.cloud:spring-cloud-consul-core:jar:1.2.1.RELEASE:compile
[INFO] |  |  +- com.ecwid.consul:consul-api:jar:1.2.2:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.8.1:compile
[INFO] |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] |  |  |  \- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-consul-discovery:jar:1.2.1.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-netflix-core:jar:1.3.4.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-starter-ribbon:jar:1.3.4.RELEASE:compile
[INFO] |  |  +- com.netflix.ribbon:ribbon:jar:2.2.2:compile
[INFO] |  |  |  +- com.netflix.ribbon:ribbon-transport:jar:2.2.2:runtime
[INFO] |  |  |  |  +- io.reactivex:rxnetty-contexts:jar:0.4.9:runtime
[INFO] |  |  |  |  \- io.reactivex:rxnetty-servo:jar:0.4.9:runtime
[INFO] |  |  |  +- javax.inject:javax.inject:jar:1:runtime
[INFO] |  |  |  \- io.reactivex:rxnetty:jar:0.4.9:runtime
[INFO] |  |  |     \- io.netty:netty-transport-native-epoll:jar:4.1.14.Final:runtime
[INFO] |  |  |        \- io.netty:netty-transport-native-unix-common:jar:4.1.14.Final:runtime
[INFO] |  |  +- com.netflix.ribbon:ribbon-core:jar:2.2.2:compile
[INFO] |  |  |  \- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  |  +- com.netflix.ribbon:ribbon-httpclient:jar:2.2.2:compile
[INFO] |  |  |  +- commons-collections:commons-collections:jar:3.2.2:runtime
[INFO] |  |  |  +- com.sun.jersey:jersey-client:jar:1.19.1:runtime
[INFO] |  |  |  |  \- com.sun.jersey:jersey-core:jar:1.19.1:runtime
[INFO] |  |  |  |     \- javax.ws.rs:jsr311-api:jar:1.1.1:runtime
[INFO] |  |  |  \- com.sun.jersey.contribs:jersey-apache-client4:jar:1.19.1:runtime
[INFO] |  |  +- com.netflix.ribbon:ribbon-loadbalancer:jar:2.2.2:compile
[INFO] |  |  |  \- com.netflix.netflix-commons:netflix-statistics:jar:0.1.1:runtime
[INFO] |  |  \- io.reactivex:rxjava:jar:1.1.10:compile
[INFO] |  \- joda-time:joda-time:jar:2.9.9:compile
[INFO] +- org.springframework.cloud:spring-cloud-sleuth-stream:jar:1.2.4.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-sleuth-core:jar:1.2.4.RELEASE:compile
[INFO] |  |  \- org.aspectj:aspectjrt:jar:1.8.10:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-stream:jar:1.2.2.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-validation:jar:1.5.6.RELEASE:compile
[INFO] |  |  +- org.springframework:spring-messaging:jar:4.3.10.RELEASE:compile
[INFO] |  |  +- org.springframework.integration:spring-integration-core:jar:4.3.11.RELEASE:compile
[INFO] |  |  |  \- org.springframework:spring-tx:jar:4.3.10.RELEASE:compile
[INFO] |  |  +- org.springframework.integration:spring-integration-jmx:jar:4.3.11.RELEASE:compile
[INFO] |  |  +- org.springframework:spring-tuple:jar:1.0.0.RELEASE:compile
[INFO] |  |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO] |  |  |     \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |  +- org.springframework.integration:spring-integration-tuple:jar:1.0.0.RELEASE:compile
[INFO] |  |  \- org.springframework.retry:spring-retry:jar:1.2.1.RELEASE:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-commons:jar:1.2.3.RELEASE:compile
[INFO] |  |  \- org.springframework.security:spring-security-crypto:jar:4.2.3.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-aop:jar:1.5.6.RELEASE:compile
[INFO] |     \- org.aspectj:aspectjweaver:jar:1.8.10:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-stream-kafka:jar:1.2.1.RELEASE:compile
[INFO] |  \- org.springframework.cloud:spring-cloud-stream-binder-kafka:jar:1.2.1.RELEASE:compile
[INFO] |     +- org.springframework.cloud:spring-cloud-stream-binder-kafka-core:jar:1.2.1.RELEASE:compile
[INFO] |     +- org.springframework.cloud:spring-cloud-stream-codec:jar:1.2.2.RELEASE:compile
[INFO] |     +- org.apache.kafka:kafka_2.11:jar:0.10.1.1:compile
[INFO] |     |  +- net.sf.jopt-simple:jopt-simple:jar:4.9:compile
[INFO] |     |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |     |  +- org.scala-lang:scala-library:jar:2.11.8:compile
[INFO] |     |  +- com.101tec:zkclient:jar:0.10:compile
[INFO] |     |  +- org.apache.zookeeper:zookeeper:jar:3.4.9:compile
[INFO] |     |  |  \- io.netty:netty:jar:3.10.5.Final:compile
[INFO] |     |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |     +- org.apache.kafka:kafka-clients:jar:0.10.1.1:compile
[INFO] |     |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |     |  \- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |     +- org.springframework.kafka:spring-kafka:jar:1.1.6.RELEASE:compile
[INFO] |     \- org.springframework.integration:spring-integration-kafka:jar:2.1.0.RELEASE:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-feign:jar:1.3.4.RELEASE:compile
[INFO] |  +- io.github.openfeign:feign-core:jar:9.4.0:compile
[INFO] |  |  \- org.jvnet:animal-sniffer-annotation:jar:1.0:compile
[INFO] |  +- io.github.openfeign:feign-slf4j:jar:9.4.0:compile
[INFO] |  |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] |  +- io.github.openfeign:feign-hystrix:jar:9.4.0:compile
[INFO] |  \- org.springframework.cloud:spring-cloud-starter-archaius:jar:1.3.4.RELEASE:compile
[INFO] |     +- com.netflix.archaius:archaius-core:jar:0.7.4:compile
[INFO] |     \- commons-configuration:commons-configuration:jar:1.10:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-zuul:jar:1.3.4.RELEASE:compile
[INFO] |  \- com.netflix.zuul:zuul-core:jar:1.3.0:compile
[INFO] |     +- commons-io:commons-io:jar:2.5:runtime
[INFO] |     +- com.netflix.servo:servo-core:jar:0.10.1:runtime
[INFO] |     |  \- com.netflix.servo:servo-internal:jar:0.10.1:runtime
[INFO] |     \- com.netflix.netflix-commons:netflix-commons-util:jar:0.1.1:runtime
[INFO] +- org.projectlombok:lombok:jar:1.16.12:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:1.5.6.RELEASE:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:1.5.6.RELEASE:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:1.5.6.RELEASE:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.2.0:test
[INFO] |  |  \- net.minidev:json-smart:jar:2.2.1:test
[INFO] |  |     \- net.minidev:accessors-smart:jar:1.1:test
[INFO] |  +- junit:junit:jar:4.12:test
[INFO] |  +- org.assertj:assertj-core:jar:2.6.0:test
[INFO] |  +- org.mockito:mockito-core:jar:1.10.19:test
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] |  +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.4.0:test
[INFO] |  +- org.springframework:spring-core:jar:4.3.10.RELEASE:compile
[INFO] |  \- org.springframework:spring-test:jar:4.3.10.RELEASE:test
[INFO] +- org.springframework.cloud:spring-cloud-starter-hystrix:jar:1.3.4.RELEASE:compile
[INFO] |  +- com.netflix.hystrix:hystrix-core:jar:1.5.12:compile
[INFO] |  |  \- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
[INFO] |  +- com.netflix.hystrix:hystrix-metrics-event-stream:jar:1.5.12:compile
[INFO] |  |  \- com.netflix.hystrix:hystrix-serialization:jar:1.5.12:runtime
[INFO] |  |     \- com.fasterxml.jackson.module:jackson-module-afterburner:jar:2.8.9:runtime
[INFO] |  \- com.netflix.hystrix:hystrix-javanica:jar:1.5.12:compile
[INFO] |     +- org.apache.commons:commons-lang3:jar:3.4:runtime
[INFO] |     \- org.ow2.asm:asm:jar:5.0.4:runtime
[INFO] +- org.springframework.cloud:spring-cloud-starter-hystrix-dashboard:jar:1.3.4.RELEASE:compile
[INFO] |  \- org.springframework.cloud:spring-cloud-netflix-hystrix-dashboard:jar:1.3.4.RELEASE:compile
[INFO] |     +- org.springframework.boot:spring-boot-starter-freemarker:jar:1.5.6.RELEASE:compile
[INFO] |     |  +- org.freemarker:freemarker:jar:2.3.26-incubating:compile
[INFO] |     |  \- org.springframework:spring-context-support:jar:4.3.10.RELEASE:compile
[INFO] |     +- org.webjars:jquery:jar:2.1.1:compile
[INFO] |     \- org.webjars:d3js:jar:3.4.11:compile
[INFO] +- org.springframework.boot:spring-boot-configuration-processor:jar:1.5.6.RELEASE:compile
[INFO] |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:compile
[INFO] +- com.company.common:grpc-client-spring-boot-starter:jar:0.1.0-SNAPSHOT:compile
[INFO] |  \- com.company.common:grpc-client-spring-boot-autoconfigure:jar:0.1.0-SNAPSHOT:compile
[INFO] +- com.company.common:event-logging-spring-boot-starter:jar:0.4.0:compile
[INFO] |  +- com.company.common:event-models:jar:0.2.0:compile
[INFO] |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.8.9:compile
[INFO] |  +- com.company.common:logging-core:jar:0.4.0:compile
[INFO] |  |  \- commons-beanutils:commons-beanutils:jar:1.8.3:compile
[INFO] |  \- com.company.common:event-logging-spring-boot-autoconfigure:jar:0.4.0:compile
[INFO] +- io.grpc:grpc-netty:jar:1.6.1:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.6.1:compile
[INFO] |  |  +- io.grpc:grpc-context:jar:1.6.1:compile
[INFO] |  |  +- com.google.errorprone:error_prone_annotations:jar:2.0.19:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] |  |  +- com.google.instrumentation:instrumentation-api:jar:0.4.3:compile
[INFO] |  |  \- io.opencensus:opencensus-api:jar:0.5.1:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.14.Final:compile
[INFO] |  |  +- io.netty:netty-codec-http:jar:4.1.14.Final:compile
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.14.Final:compile
[INFO] |  |  \- io.netty:netty-handler:jar:4.1.14.Final:compile
[INFO] |  |     \- io.netty:netty-buffer:jar:4.1.14.Final:compile
[INFO] |  \- io.netty:netty-handler-proxy:jar:4.1.14.Final:compile
[INFO] |     +- io.netty:netty-transport:jar:4.1.14.Final:compile
[INFO] |     |  \- io.netty:netty-resolver:jar:4.1.14.Final:compile
[INFO] |     \- io.netty:netty-codec-socks:jar:4.1.14.Final:compile
[INFO] \- com.company.common:logging:jar:1.0.0:compile
marcingrzejszczak commented 7 years ago

Without a sample to reproduce this problem it's going to be hard to help you. As a precaution, what I did is I've added AutoConfigureAfter the Spring Integration autoconfiguration. Hope it helps

gdong42 commented 7 years ago

I suspect it is the same issue with https://github.com/spring-cloud/spring-cloud-netflix/issues/912

Why is this closed? The problem is still there. I'll try if I can extract things into a sample.

marcingrzejszczak commented 7 years ago

I wrote why I closed it. If you say it's still there then I'll reopen it and wait for the sample

gdong42 commented 7 years ago

@marcingrzejszczak I tracked down the root cause - it was due to a thirdparty dependency. In its @Bean definition, it requires a bean dependency of Tracer, in order to add tracing ability, which causes sleuthTracer (and its dependencies including sleuthStreamSpanReporter) being early instantiated.

I'll open issue in that thirdparty project. Thanks for your help!

Regards

marcingrzejszczak commented 7 years ago

Glad that you tracked it down! Which dependency is it BTW?

gdong42 commented 7 years ago

@marcingrzejszczak above referencing one. It's a gRPC and Spring Boot integration starter.

The code snippet is here:

https://github.com/yidongnan/grpc-spring-boot-starter/blob/dbde8a7d78b180b8b0e0ccf2f0b136808ed3a707/grpc-client-spring-boot-autoconfigure/src/main/java/net/devh/springboot/autoconfigure/grpc/client/GrpcClientAutoConfiguration.java#L73

liwei-link commented 6 years ago

I got the same problem. StreamSpanReporter.poll() is never called. This is my pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>demo</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.13.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <spring-cloud.version>Edgware.SR3</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-sleuth-stream</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>
liwei-link commented 6 years ago

Here is my sample https://github.com/leewill1120/demo.git

marcingrzejszczak commented 6 years ago

How am I supposed to test your sample? You've provided a single app that calls another app that doesn't exist. Also can you check if the problem is still there when using sleuth snapshots. We fixed an issue related to proxies with this issue - https://github.com/spring-cloud/spring-cloud-sleuth/issues/932 .

liwei-link commented 6 years ago

I tried again with snapshot(1.3.4.BUILD-SNAPSHOT), It works! Thank you.