search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.26k stars 3.79k forks source link

Include exemplars in OpenCensus instrumentation #7505

Open aabmass opened 3 years ago

aabmass commented 3 years ago

Is your feature request related to a problem?

The OpenCensus GRPC instrumentation does not attach exemplar spans along with the distribution stats it collects.

Exemplars are example points that may be used to annotate aggregated distribution values. They are metadata that gives information about a particular value added to a Distribution bucket, such as a trace ID that was active when a value was added. They may contain further information, such as a example values and timestamps, origin, etc.

Google cloud monitoring can interpret and display these exemplars to link metrics to example spans.

Describe the solution you'd like

Add exemplars to grcp-census instrumentation directly near this code: https://github.com/grpc/grpc-java/blob/020325617129b6ae9b25da1126ba0f98b044bb1b/census/src/main/java/io/grpc/census/CensusStatsModule.java#L647

Adding something like this should work:

ExemplarUtils.putSpanContextAttachments(measureMap, tracer.getCurrentSpan().getContext());

Describe alternatives you've considered

I don't think there is a way to do this without updating the grpc-census instrumentation code. AFAIK adding the exemplar to the MeasureMap directly is the only way to do it.

Additional context

I'm not sure how to get the OpenCensus Tracer for tracer.getCurrentSpan() inside of CensusStatsModule. It may also be beneficial to check if the current span is being sampled before attaching it to the MeasureMap.

cc @nilebox

aabmass commented 3 years ago

I am happy to submit a PR for this 🙂

dapengzhang0 commented 3 years ago

@aabmass Does it need add extra dependencies to io.grpc:grpc-census? Currently we only depend on io.opencensus:opencensus-api and io.opencensus:opencensus-contrib-grpc-metrics

aabmass commented 3 years ago

@dapengzhang0 I believe it needs io.opencensus:opencensus-contrib-exemplar-util which is pretty small

aabmass commented 3 years ago

@dapengzhang0 would you be open to a PR for this?

dapengzhang0 commented 3 years ago

would you be open to a PR for this?

@aabmass We would like to know why we/google cloud want this feature before making a PR. Would you send some more detail feature request doc internally and let grpc-java-team@ review?

ejona86 commented 3 years ago

I've re-read some internal conversation and looked at the open PR and I think I now understand what an exemplar is and now the details make more sense.

Since the exemplars are only attached to the measure map for sampled traces, that means the CPU and memory costs will be very low. And it looks like a very useful feature. I think that means this is safe to enable for everyone without requiring an option. That makes this a lot easier to add. I think this stalled earlier because there was talk of it needing to be configurable and "configurable OpenCensus" is still an open issue.

There are still some implementation difficulties to consider, since this is the first time stats and tracing interact with each other.