search

spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
704 stars 693 forks source link

Traces not propagated to Cloud Trace, StackdriverTraceAutoConfiguration fails with timeout #2671

Closed irenavy closed 3 years ago

irenavy commented 3 years ago

Traces are not being propagated to Cloud trace from my spring boot application. StackdriverTraceAutoConfiguration fails with timeout exception -

"Error when performing Stackdriver AsyncReporter health check. java.lang.IllegalStateException: timeout waiting for onClose. timeoutMs=5000, resultSet=false at zipkin2.reporter.stackdriver.internal.AwaitableUnaryClientCallListener.await(AwaitableUnaryClientCallListener.java:49) at zipkin2.reporter.stackdriver.internal.UnaryClientCall.doExecute(UnaryClientCall.java:50) at zipkin2.Call$Base.execute(Call.java:380) at zipkin2.reporter.stackdriver.StackdriverSender.check(StackdriverSender.java:183) at zipkin2.reporter.AsyncReporter$BoundedAsyncReporter.check(AsyncReporter.java:333) at org.springframework.cloud.gcp.autoconfigure.trace.StackdriverTraceAutoConfiguration.stackdriverReporter(StackdriverTraceAutoConfiguration.java:166) at org.springframework.cloud.gcp.autoconfigure.trace.StackdriverTraceAutoConfiguration$$EnhancerBySpringCGLIB$$44773908.CGLIB$stackdriverReporter$1()

Spring boot version: 2.3.2.RELEASE Spring Cloud GCP version: 1.2.4.RELEASE

There is no option available for configuring server timeout. It is failing during autoconfiguration and I am not able to identify the rootcause.

meltsufin commented 3 years ago

Some context here: https://github.com/openzipkin/zipkin-gcp/pull/102

Maybe there is an issue with credentials. Are you able to upgrade to Spring Cloud GCP 2.x to try the latest version?

irenavy commented 3 years ago

Thanks @meltsufin , I added roles/cloudtrace.agent to service account, which resolved timeout error. But traces are still not sent to Cloud trace.

Traces are being sent to stackdriver logging -

image

Some additional details - the microservice is a pubsub subscriber and we are using spring cloud stream binder for consuming and publishing messages to topics. It is deployed on GKE and uses istio proxy.

irenavy commented 3 years ago

As per documentation,

Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and subscribe events.

Basically, I want to capture traces and spans for these events. Will spring-cloud-gcp-starter-trace dependency suffice or I need any additional dependencies to make it work with PubSub calls? I could not find any example for tracing pubsub events.

meltsufin commented 3 years ago

Pub/Sub is not fully supported with Sleuth yet. You can get some tracing if you're using Spring Integration with Pub/Sub, but not the Pub/Sub Template. There is an issue for adding the support: https://github.com/openzipkin/zipkin-gcp/issues/190.

irenavy commented 3 years ago

So PubSub events will not able be propagated to Cloud Trace automatically. Can you please help me with below queries:

Should it work with custom spans if I include grpc dependencies

Spring Cloud Sleuth provides instrumentation for gRPC through TraceGrpcAutoConfiguration

Can you please suggest a way of sending traces to stackdriver trace when using cloud stream binder? I am using Cloud sleuth for distributed tracing and it works fine with stackdriver logging i.e for each request new traceid is generated and same is propagated to next microservice. I am not sure if I can use alternatives like openTelemetry along with sleuth for exporting traces to stackdriver trace.

meltsufin commented 3 years ago

Stream binder uses Spring Integration under the hood. So, it might work. Can you give it a try and let us know? Regarding TraceGrpcAutoConfiguration, I believe it's only relevant to when you're creating your own gRPC service.