Clarify Sleuth support

[asarkar]

It's not at all clear from the documentation what is traced in the presence of Spring Cloud Sleuth. Are tracing headers added to all client-server calls? How's that helpful if the server doesn't return those back? Please update the docs to clarify, preferably using an image like this

[ST-DDT]

I'm not overly familiar with sleuth, so I appreciate any help.

[asarkar]

@ST-DDT Are you saying that the code claims to support Sleuth, but you don't know what it does?

[ST-DDT]

This sounds a bit offensive to me. But maybe i just used the wrong wording.

I know how it works, but I dont know how (and also what) to describe it in a way thats adds more value to it than a link to sleuth's docs itself.

Setup howto: Add sleuth similar to your http app Add the grpc-instrumentation as dependency Well, thats all you have to do.

Which data are transfered for which purpose? sleuth/brave/zipkin's docs are centered on that topic, so why should I repeat it?!

[asarkar]

I don't know why you were offended, but some people are more sensitive than others. It was obviously not my intent to offend you.

Sleuth works by adding metadata to remote calls. For HTTP, it adds headers; I assume that's what it does for gRPC. What I'm trying to understand here is what does it mean to enable Sleuth dependency. For bidirectional streaming, the client will send several requests, and receive several response. Unless someone owns both client and server side code, and adds Sleuth to the build on both ends, I don't understand how Sleuth will be able to inject metadata. I also don't understand what this library does when Sleuth is on the classpath.

Is my question clearer now? I can, of course, look in the code, but I think this particular section of the docs could use some attention.

[ST-DDT]

(I had the feeling we/this library were being accused of making a false claim.) If that wasn't intended, then I don't mind.

Aside from sleuth you also need brave-instrumentation-grpc. That library contains all the relevant logic. This library only creates the few relevant beans.

Does the following link help you to understand how it works? https://github.com/openzipkin/brave/tree/master/instrumentation/grpc#message-processing-callback-context

[asarkar]

So it turns out that Sleuth gRPC tracing is completely broken. I opened https://github.com/spring-cloud/spring-cloud-sleuth/issues/1749, which was closed because “it wasn’t their problem”. I then opened https://github.com/openzipkin/zipkin-support/issues/40, which was closed because “it is not easy to implement”. I then opened https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1313, which was implemented and seems to be working.

In short, the only library that traces gRPC properly is OpenTelemetry, and it isn’t released yet.

[ST-DDT]

AFAICT its only broken for (Kotlin) coroutines. However, the fact that they have such a problem seems somewhat suspicious to me. I assume that there is simply some kind of (context) bridge missing.

I don't have time to have an in depth look into this anytime soon. Would it be possible for you to write a minimalistic (Java based) test for this? Running it in a Kotlin JRE is probably not an issue. If a strictly java source code based test doesn't work, then a Kotlin one is fine too, as long as I can use my usual debugging tools. Then maybe I can figure out where their tracing context is lost, which is the first half of fixing the issue.

[asarkar]

It could be that Kotlin coroutine is making things worse, but the fundamental issue is the Brave design that context isn't propagated using gRPC context but their own. OpenTelemetry was going down that route, until they listened to my suggestion, and simply used gRPC context instead. They did nothing special for coroutines, but it works like a charm.

You may close this ticket if you feel like; I have little interest in supporting libraries that simply brush off user problems (Sleuth and Brave).

As an aside, I do believe that cross-cutting concerns like tracing and metrics should be separate from the core library, because people may not be using the same third-party libraries that you assumed they would (like Sleuth or Micrometer).

[anjeyy]

I was about to open up an issue, but i won't since i found this one still open.

I have troubles adding sleuth logging via zipkin to my grpc-sample project. Spend the whole day trying to bind the logging, but i can not get it to work, relly frustrating. Everything else besides this is working nice so far, so thank you.

Please if anyone can give me an idiot proof example how to bind it in my project, i appreciate it! Thank you.

I am getting following error by declaring dependencies as you stated - https://yidongnan.github.io/grpc-spring-boot-starter/en/brave.html

Full source code is linked above in my first setence.

PR with source code: https://github.com/anjeyy/grpc-sample/pull/7

2020-12-11 17:05:43.735  WARN [,,,] 10500 --- [           main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationPropertiesBeans' defined in class path resource [org/springframework/cloud/autoconfigure/ConfigurationPropertiesRebinderAutoConfiguration.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.context.properties.ConfigurationPropertiesBeans] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@368239c8]
2020-12-11 17:05:43.743  INFO [,,,] 10500 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-12-11 17:05:43.761 ERROR [,,,] 10500 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationPropertiesBeans' defined in class path resource [org/springframework/cloud/autoconfigure/ConfigurationPropertiesRebinderAutoConfiguration.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.context.properties.ConfigurationPropertiesBeans] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@368239c8]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:586) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:531) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:244) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:767) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:572) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:144) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:212) ~[spring-cloud-context-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:117) ~[spring-cloud-context-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:74) ~[spring-cloud-context-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:170) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:148) ~[spring-context-5.3.1.jar:5.3.1]
    at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:82) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:63) ~[spring-boot-2.4.0.jar:2.4.0]
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[na:na]
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:117) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:111) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:62) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:362) ~[spring-boot-2.4.0.jar:2.4.0]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:320) ~[spring-boot-2.4.0.jar:2.4.0]
    at com.github.anjeyy.GrpcServerApplication.main(GrpcServerApplication.java:15) ~[classes/:na]
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.context.properties.ConfigurationPropertiesBeans] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@368239c8]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:481) ~[spring-core-5.3.1.jar:5.3.1]
    at org.springframework.util.ReflectionUtils.doWithLocalMethods(ReflectionUtils.java:321) ~[spring-core-5.3.1.jar:5.3.1]
    at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.buildPersistenceMetadata(PersistenceAnnotationBeanPostProcessor.java:417) ~[spring-orm-5.3.1.jar:5.3.1]
    at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.findPersistenceMetadata(PersistenceAnnotationBeanPostProcessor.java:388) ~[spring-orm-5.3.1.jar:5.3.1]
    at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(PersistenceAnnotationBeanPostProcessor.java:335) ~[spring-orm-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyMergedBeanDefinitionPostProcessors(AbstractAutowireCapableBeanFactory.java:1100) ~[spring-beans-5.3.1.jar:5.3.1]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:583) ~[spring-beans-5.3.1.jar:5.3.1]
    ... 29 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/springframework/boot/context/properties/ConfigurationBeanFactoryMetadata
    at java.base/java.lang.Class.getDeclaredMethods0(Native Method) ~[na:na]
    at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166) ~[na:na]
    at java.base/java.lang.Class.getDeclaredMethods(Class.java:2309) ~[na:na]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:463) ~[spring-core-5.3.1.jar:5.3.1]
    ... 35 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.context.properties.ConfigurationBeanFactoryMetadata
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    ... 39 common frames omitted

[ST-DDT]

@anjeyy You might have run into this issue: https://github.com/spring-projects/spring-boot/issues/19860 (org.springframework.boot.context.properties.ConfigurationBeanFactoryMetadata has been removed/replaced in spring-boot 2.4.0)

Maybe you need a newer version of spring-cloud-starter-sleuth?