Closed guerricmerleHUG closed 5 years ago
I've run into the same issue. I've been able to verify that the services called by the gateway do not receive any of the X-B3-*
headers. I am seeing calls to brave.propagation.B3Propagation.B3Injector.inject(TraceContext traceContext, C carrier)
. This explains why the PatientService
in the example has a separate trace.
I've also noticed that when I'm using StripPrefix=1
, the http.path
for the span is updated with the stripped path. It feels like the creation of a child span isn't occurring correctly.
We ran into the same issue with Greenwich.M2
. Can also confirm on Greenwich.M3
.
We used httpbin.org/headers
as upstream. The headers are definitely missing.
application.yml
:
spring:
sleuth:
enabled: true
sampler.probability: 1
zipkin:
enabled: true
base-url: replace_me
Can confirm on Greenwich.RC2
logger_name reactor.netty.http.server.AccessLog
doesn't have traceId
We are blocked by https://github.com/reactor/reactor-netty/issues/558
@spencergibb suggested to work this around in Sleuth via the Gateway interfaces. It's been done via https://github.com/spring-cloud/spring-cloud-sleuth/commit/ec60f3a3a57721d78fff5672bc542673a94fa359 . Can anyone wait for the snapshots to be built and confirm that indeed everything is working fine? I double checked it in the Brewery and it seems to be working fine
I'll check and answer as soon as snapshot build will be ready. 2.1.0.BUILD-SNAPSHOT/ 25-Jul-2018 09:18
Awesome. Give it 3 more minutes and the snapshots should be fully uploaded
Should your fix add tracing to access log?
So I have such configuration in my gateway:
springBootVersion = '2.1.1.RELEASE'
springCloudVersion = 'Greenwich.RC2'
dependencyManagement {
imports {
mavenBom "org.springframework.boot:spring-boot-starter-parent:${springBootVersion}"
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
}
}
dependencies {
compile ('org.springframework.cloud:spring-cloud-starter-netflix-eureka-client')
compile('org.springframework.boot:spring-boot-starter-webflux')
compile ('org.springframework.boot:spring-boot-starter-actuator')
compile('org.springframework.cloud:spring-cloud-starter-gateway')
compile('org.springframework.cloud:spring-cloud-starter-sleuth:2.1.0.BUILD-SNAPSHOT')
compile('org.springframework.cloud:spring-cloud-sleuth-core:2.1.0.BUILD-SNAPSHOT')
compile('org.springframework.cloud:spring-cloud-config-client')
compile('org.springframework.cloud:spring-cloud-starter-bus-amqp')
}
Libs in boot jar: But access log doesn't have tracing.
Nope - there's a separate issue for that https://github.com/spring-cloud/spring-cloud-sleuth/issues/1131
Ok, I got it. Give me some time and I'll try to reproduce and verify issue of @guerricm
@marcingrzejszczak with same config gateway responds me 500 every time with snapshot sleuth. In logs no errors, only
2019-01-09 10:56:47.805 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] o.s.w.s.adapter.HttpWebHandlerAdapter : [6072d2d9] HTTP GET "/first/tests"
2019-01-09 10:56:47.807 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] o.s.c.s.instrument.web.TraceWebFilter : Received a request to uri [/first/tests]
2019-01-09 10:56:47.808 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] o.s.c.s.instrument.web.TraceWebFilter : Handled receive of span NoopSpan(83081568d0a8d04f/83081568d0a8d04f)
2019-01-09 10:56:47.814 DEBUG [gateway,83081568d0a8d04f,83081568d0a8d04f,false] 1 --- [or-http-epoll-4] .c.s.i.w.c.TraceRequestHttpHeadersFilter : Will instrument the HTTP request headers
2019-01-09 10:56:47.815 DEBUG [gateway,83081568d0a8d04f,83081568d0a8d04f,false] 1 --- [or-http-epoll-4] o.s.c.s.instrument.web.TraceWebFilter : Handled send of NoopSpan(83081568d0a8d04f/83081568d0a8d04f)
2019-01-09 10:56:47.821 DEBUG [gateway,83081568d0a8d04f,83081568d0a8d04f,false] 1 --- [or-http-epoll-4] o.s.w.s.adapter.HttpWebHandlerAdapter : [6072d2d9] Completed 500 INTERNAL_SERVER_ERROR
With RC sleuth everything is OK
I might have fixed that in the meantime. Can you check again?
The same
2019-01-09 11:44:16.361 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] o.s.c.s.instrument.web.TraceWebFilter : Received a request to uri [/first/tests]
2019-01-09 11:44:16.362 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] o.s.c.s.instrument.web.TraceWebFilter : Handled receive of span NoopSpan(79257c8f2b63ebb2/79257c8f2b63ebb2)
2019-01-09 11:44:16.369 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] o.s.c.g.h.RoutePredicateHandlerMapping : Route matched: first
2019-01-09 11:44:16.370 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] o.s.c.g.h.RoutePredicateHandlerMapping : Mapping [Exchange: GET http://localhost/first/tests] to Route{id='first', uri=lb://first, order=0, predicate=org.springframework.cloud.gateway.support.ServerWebExchangeUtils$$Lambda$1003/865249158@4f5ecdeb, gatewayFilters=[]}
2019-01-09 11:44:16.371 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] o.s.c.g.h.RoutePredicateHandlerMapping : [e4dbb76a] Mapped to org.springframework.cloud.gateway.handler.FilteringWebHandler@5c77053b
2019-01-09 11:44:16.372 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] o.s.c.g.handler.FilteringWebHandler : Sorted gatewayFilterFactories: [OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter@7807ac2c}, order=-2147482648}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@f19c9d2}, order=-1}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter@a77614d}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter@5398edd0}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@b91d8c4}, order=10000}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.LoadBalancerClientFilter@741b3bc3}, order=10100}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter@4cc76301}, order=2147483646}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@5118388b}, order=2147483647}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter@4b6166aa}, order=2147483647}]
2019-01-09 11:44:16.375 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] .c.s.i.w.c.TraceRequestHttpHeadersFilter : Will instrument the HTTP request headers
2019-01-09 11:44:16.376 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] o.s.c.s.instrument.web.TraceWebFilter : Handled send of NoopSpan(79257c8f2b63ebb2/79257c8f2b63ebb2)
2019-01-09 11:44:16.383 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] r.n.channel.ChannelOperationsHandler : [id: 0xe4dbb76a, L:/10.255.3.103:8881 - R:/10.255.0.2:53358] Writing object DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 500 Internal Server Error
Content-Type: text/html
Content-Length: 281
2019-01-09 11:44:16.385 DEBUG [gateway,79257c8f2b63ebb2,79257c8f2b63ebb2,false] 1 --- [or-http-epoll-4] r.n.channel.ChannelOperationsHandler : [id: 0xe4dbb76a, L:/10.255.3.103:8881 - R:/10.255.0.2:53358] Writing object
2019-01-09 11:44:16.388 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] r.n.http.server.HttpServerOperations : [id: 0xe4dbb76a, L:/10.255.3.103:8881 - R:/10.255.0.2:53358] Last HTTP response frame
2019-01-09 11:44:16.390 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] r.n.channel.ChannelOperationsHandler : [id: 0xe4dbb76a, L:/10.255.3.103:8881 - R:/10.255.0.2:53358] Writing object EmptyLastHttpContent
2019-01-09 11:44:16.391 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] r.n.http.server.HttpServerOperations : [id: 0xe4dbb76a, L:/10.255.3.103:8881 - R:/10.255.0.2:53358] Decreasing pending responses, now 0
2019-01-09 11:44:16.393 DEBUG [gateway,,,] 1 --- [or-http-epoll-4] r.n.channel.ChannelOperationsHandler : [id: 0xe4dbb76a, L:/10.255.3.103:8881 - R:/10.255.0.2:53358] No ChannelOperation attached. Dropping: EmptyLastHttpContent
I can prepare sample project for you if needed
I see that my log doesn't contain
log.debug("Client span " + span + " created for the request. New headers are "
+ builder.build().getHeaders().toSingleValueMap());
So possible that I have old snapshot jar in my classpath(but I cleared all gradle\maven local caches before build) or error in these lines
ServerHttpRequest.Builder builder = exchange.getRequest().mutate();
Span span = this.handler.handleSend(this.injector, builder);
i checked too with
org.springframework.cloud:spring-cloud-starter-sleuth:2.1.0.BUILD-SNAPSHOT
and
org.springframework.cloud:spring-cloud-sleuth-core:2.1.0.BUILD-SNAPSHOT
and i have the same error as @RostislavReznichenko : Completed 500 INTERNAL_SERVER_ERROR
2019-01-09 13:33:28.220 DEBUG [gateway-service,,,] 4652 --- [ctor-http-nio-2] o.s.c.s.instrument.web.TraceWebFilter : Received a request to uri [/PATIENT-SERVICE/patients/97005400/eds/1]
2019-01-09 13:33:28.228 DEBUG [gateway-service,,,] 4652 --- [ctor-http-nio-2] o.s.c.s.instrument.web.TraceWebFilter : Handled receive of span RealSpan(1963538efe06dea9/1963538efe06dea9)
2019-01-09 13:33:28.299 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_EPISODEOFCARE-SERVICE applying {pattern=/EPISODEOFCARE-SERVICE/**} to Path
2019-01-09 13:33:28.308 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_EPISODEOFCARE-SERVICE applying filter {regexp=/EPISODEOFCARE-SERVICE/(?<remaining>.*), replacement=/${remaining}} to RewritePath
2019-01-09 13:33:28.311 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition matched: CompositeDiscoveryClient_EPISODEOFCARE-SERVICE
2019-01-09 13:33:28.312 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_CONFIGSERVER applying {pattern=/CONFIGSERVER/**} to Path
2019-01-09 13:33:28.312 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_CONFIGSERVER applying filter {regexp=/CONFIGSERVER/(?<remaining>.*), replacement=/${remaining}} to RewritePath
2019-01-09 13:33:28.313 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition matched: CompositeDiscoveryClient_CONFIGSERVER
2019-01-09 13:33:28.313 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_PATIENT-SERVICE applying {pattern=/PATIENT-SERVICE/**} to Path
2019-01-09 13:33:28.314 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_PATIENT-SERVICE applying filter {regexp=/PATIENT-SERVICE/(?<remaining>.*), replacement=/${remaining}} to RewritePath
2019-01-09 13:33:28.314 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition matched: CompositeDiscoveryClient_PATIENT-SERVICE
2019-01-09 13:33:28.314 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_GATEWAY-SERVICE applying {pattern=/GATEWAY-SERVICE/**} to Path
2019-01-09 13:33:28.315 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition CompositeDiscoveryClient_GATEWAY-SERVICE applying filter {regexp=/GATEWAY-SERVICE/(?<remaining>.*), replacement=/${remaining}} to RewritePath
2019-01-09 13:33:28.315 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.r.RouteDefinitionRouteLocator : RouteDefinition matched: CompositeDiscoveryClient_GATEWAY-SERVICE
2019-01-09 13:33:28.321 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.h.RoutePredicateHandlerMapping : Route matched: CompositeDiscoveryClient_PATIENT-SERVICE
2019-01-09 13:33:28.321 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.h.RoutePredicateHandlerMapping : Mapping [Exchange: GET http://localhost/PATIENT-SERVICE/patients/97005400/eds/1] to Route{id='CompositeDiscoveryClient_PATIENT-SERVICE', uri=lb://PATIENT-SERVICE, order=0, predicate=org.springframework.cloud.gateway.support.ServerWebExchangeUtils$$Lambda$802/1214971396@411d717b, gatewayFilters=[OrderedGatewayFilter{delegate=org.springframework.cloud.gateway.filter.factory.RewritePathGatewayFilterFactory$$Lambda$804/682925415@2aa76c85, order=1}]}
2019-01-09 13:33:28.321 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.h.RoutePredicateHandlerMapping : [a1563cee] Mapped to org.springframework.cloud.gateway.handler.FilteringWebHandler@1a1ba3f8
2019-01-09 13:33:28.322 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.g.handler.FilteringWebHandler : Sorted gatewayFilterFactories: [OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter@364fd4ae}, order=-2147482648}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter@385dfb63}, order=-1}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter@459003a0}, order=0}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter@3b481bf5}, order=0}, OrderedGatewayFilter{delegate=org.springframework.cloud.gateway.filter.factory.RewritePathGatewayFilterFactory$$Lambda$804/682925415@2aa76c85, order=1}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter@245253d8}, order=10000}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.LoadBalancerClientFilter@623e0631}, order=10100}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter@d325518}, order=2147483646}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter@359066bc}, order=2147483647}, OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter@12417468}, order=2147483647}]
2019-01-09 13:33:28.334 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] s.c.a.AnnotationConfigApplicationContext : Refreshing SpringClientFactory-PATIENT-SERVICE
2019-01-09 13:33:28.334 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.context.annotation.internalConfigurationAnnotationProcessor'
2019-01-09 13:33:28.353 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.boot.autoconfigure.condition.BeanTypeRegistry'
2019-01-09 13:33:28.373 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'propertySourcesPlaceholderConfigurer'
2019-01-09 13:33:28.374 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.context.event.internalEventListenerProcessor'
2019-01-09 13:33:28.374 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.boot.context.properties.ConfigurationBeanFactoryMetadata'
2019-01-09 13:33:28.374 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.context.event.internalEventListenerFactory'
2019-01-09 13:33:28.374 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.context.annotation.internalAutowiredAnnotationProcessor'
2019-01-09 13:33:28.374 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.context.annotation.internalCommonAnnotationProcessor'
2019-01-09 13:33:28.374 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor'
2019-01-09 13:33:28.375 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'eurekaRibbonClientConfiguration'
2019-01-09 13:33:28.377 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.e.PropertySourcesPropertyResolver : Found key 'ribbon.client.name' in PropertySource 'ribbon' with value of type String
2019-01-09 13:33:28.377 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.e.PropertySourcesPropertyResolver : Found key 'ribbon.client.name' in PropertySource 'environmentProperties' with value of type String
2019-01-09 13:33:28.419 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'propertyPlaceholderAutoConfiguration'
2019-01-09 13:33:28.419 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonClientConfiguration'
2019-01-09 13:33:28.420 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.e.PropertySourcesPropertyResolver : Found key 'ribbon.client.name' in PropertySource 'ribbon' with value of type String
2019-01-09 13:33:28.420 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.e.PropertySourcesPropertyResolver : Found key 'ribbon.client.name' in PropertySource 'environmentProperties' with value of type String
2019-01-09 13:33:28.420 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonPing'
2019-01-09 13:33:28.422 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonClientConfig'
2019-01-09 13:33:28.437 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonPing' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.441 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonServerList'
2019-01-09 13:33:28.442 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonServerList' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.445 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'serverIntrospector'
2019-01-09 13:33:28.452 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.cloud.commons.httpclient.HttpClientConfiguration$ApacheHttpClientConfiguration'
2019-01-09 13:33:28.452 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.cloud.commons.httpclient.HttpClientConfiguration'
2019-01-09 13:33:28.453 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.cloud.netflix.ribbon.apache.HttpClientRibbonConfiguration$ApacheHttpClientConfiguration'
2019-01-09 13:33:28.456 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'httpClientConnectionManager'
2019-01-09 13:33:28.457 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'httpClientConnectionManager' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.457 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'httpClientConnectionManager' via factory method to bean named 'connManFactory'
2019-01-09 13:33:28.477 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'httpClient'
2019-01-09 13:33:28.479 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'httpClient' via factory method to bean named 'apacheHttpClientFactory'
2019-01-09 13:33:28.479 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'httpClient' via factory method to bean named 'httpClientConnectionManager'
2019-01-09 13:33:28.479 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'httpClient' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.494 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'org.springframework.cloud.netflix.ribbon.apache.HttpClientRibbonConfiguration'
2019-01-09 13:33:28.494 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.e.PropertySourcesPropertyResolver : Found key 'ribbon.client.name' in PropertySource 'ribbon' with value of type String
2019-01-09 13:33:28.494 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.e.PropertySourcesPropertyResolver : Found key 'ribbon.client.name' in PropertySource 'environmentProperties' with value of type String
2019-01-09 13:33:28.495 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonLoadBalancingHttpClient'
2019-01-09 13:33:28.497 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonLoadBalancer'
2019-01-09 13:33:28.498 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonServerListFilter'
2019-01-09 13:33:28.499 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonServerListFilter' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.506 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonRule'
2019-01-09 13:33:28.506 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonRule' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.515 INFO [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] c.netflix.config.ChainedDynamicProperty : Flipping property: PATIENT-SERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-01-09 13:33:28.517 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonServerListUpdater'
2019-01-09 13:33:28.517 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonServerListUpdater' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.519 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancer' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.519 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancer' via factory method to bean named 'ribbonServerList'
2019-01-09 13:33:28.519 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancer' via factory method to bean named 'ribbonServerListFilter'
2019-01-09 13:33:28.519 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancer' via factory method to bean named 'ribbonRule'
2019-01-09 13:33:28.519 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancer' via factory method to bean named 'ribbonPing'
2019-01-09 13:33:28.519 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancer' via factory method to bean named 'ribbonServerListUpdater'
2019-01-09 13:33:28.551 INFO [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] c.n.u.concurrent.ShutdownEnabledTimer : Shutdown hook installed for: NFLoadBalancer-PingTimer-PATIENT-SERVICE
2019-01-09 13:33:28.552 INFO [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] c.netflix.loadbalancer.BaseLoadBalancer : Client: PATIENT-SERVICE instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=PATIENT-SERVICE,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2019-01-09 13:33:28.558 INFO [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] c.n.l.DynamicServerListLoadBalancer : Using serverListUpdater PollingServerListUpdater
2019-01-09 13:33:28.586 INFO [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] c.netflix.config.ChainedDynamicProperty : Flipping property: PATIENT-SERVICE.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2019-01-09 13:33:28.588 INFO [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] c.n.l.DynamicServerListLoadBalancer : DynamicServerListLoadBalancer for client PATIENT-SERVICE initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=PATIENT-SERVICE,current list of Servers=[cih-8505:8002],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:1; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;]
},Server stats: [[Server:cih-8505:8002; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 01:00:00 CET 1970; First connection made: Thu Jan 01 01:00:00 CET 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@73df7dc0
2019-01-09 13:33:28.589 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'retryHandler'
2019-01-09 13:33:28.590 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'retryHandler' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.591 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancingHttpClient' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.591 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancingHttpClient' via factory method to bean named 'serverIntrospector'
2019-01-09 13:33:28.591 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancingHttpClient' via factory method to bean named 'ribbonLoadBalancer'
2019-01-09 13:33:28.591 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancingHttpClient' via factory method to bean named 'retryHandler'
2019-01-09 13:33:28.591 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancingHttpClient' via factory method to bean named 'httpClient'
2019-01-09 13:33:28.600 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Creating shared instance of singleton bean 'ribbonLoadBalancerContext'
2019-01-09 13:33:28.600 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancerContext' via factory method to bean named 'ribbonLoadBalancer'
2019-01-09 13:33:28.601 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancerContext' via factory method to bean named 'ribbonClientConfig'
2019-01-09 13:33:28.601 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.b.f.s.DefaultListableBeanFactory : Autowiring by type from bean name 'ribbonLoadBalancerContext' via factory method to bean named 'retryHandler'
2019-01-09 13:33:28.636 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] .c.s.i.w.c.TraceRequestHttpHeadersFilter : Will instrument the HTTP request headers
2019-01-09 13:33:28.644 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.c.s.instrument.web.TraceWebFilter : Handled send of RealSpan(1963538efe06dea9/1963538efe06dea9)
2019-01-09 13:33:28.679 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : [a1563cee] Resolved [UnsupportedOperationException: null] for HTTP GET /PATIENT-SERVICE/patients/97005400/eds/1
2019-01-09 13:33:28.685 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.core.codec.CharSequenceEncoder : [a1563cee] Writing "<html><body><h1>Whitelabel Error Page</h1><p>This application has no configured error view, so you (truncated)...
2019-01-09 13:33:28.705 DEBUG [gateway-service,1963538efe06dea9,1963538efe06dea9,true] 4652 --- [ctor-http-nio-2] o.s.w.s.adapter.HttpWebHandlerAdapter : [a1563cee] Completed 500 INTERNAL_SERVER_ERROR```
Yeah i think a sample will be needed. Can you upload one for me please?
@marcingrzejszczak, here is it https://github.com/eugenekainara/scg
@RostislavReznichenko Is this issue fixed. I have exactly the same issue with 2.1.0.RELEASE
@thekalinga for us, the issue is fixed with SpringCloud Greewich.RELEASE and SpringBoot 2.1.3.RELEASE
@guerricm Did some digging
This issue is still open https://github.com/reactor/reactor-netty/issues/558
And there is a pull request https://github.com/reactor/reactor-netty/pull/622 thats is yet to be merged
I dont think the issue is fixed w.r.t WebClient + Reactory Netty combo. Can you please double check
I was using Greenwich.RELEASE
before. I now switched to Greenwich.SR1
based on @marcingrzejszczak 's advice. The header propagation now works
@marcingrzejszczak @guerricm Thanks!!
Cause SR1 is a newer release train than RELEASE. SO in SR1 you have a bug fix for this issue
@marcingrzejszczak Hi, sorry, I misunderstood the difference between release vs SR1. Thanks , now it works.
Bug report spring-cloud-version: Greenwich.M3
We used Spring Cloud Gateway and we starting to integrate Sleuth / Zipkin.
We make this workflow :
and we have activate DEBUG logs for Sleuth in SC, PS and EOCS apps
logging.level.org.springframework.cloud.sleuth: DEBUG
In Postman or Chrome, we call http://localhost/patient/97005400/with-eds this is pass througth Gateway, PatientService and EpisodeOfCareServicelogs are : Gateway :
PatientService :
EpisodeOfCareService :
As you can see, traceId are différents between Gateway and PatientService/EpisodeOfCareService. So in Zipkin, there is no correlation :
gateway config : gateway-build.gradle.txt application.yml
Patient : patient-build.gradle.txt application.yml
EOCS : eds-build.gradle.txt application.yml (same as patient-service)
What are we doing wrong ? Thanks for your help