Missing grpc status code label in trace metrics

DataDog / dd-trace-go

Datadog Go Library including APM tracing, profiling, and security monitoring.

https://docs.datadoghq.com/tracing/

Other

665 stars 437 forks source link

Open juanzolotoochin opened 1 month ago

juanzolotoochin commented 1 month ago

I'm using UnaryServerInterceptor from "gopkg.in/DataDog/dd-trace-go.v1/contrib/google.golang.org/grpc".

This generates a metric trace.grpc.server.hits which is the main metric I have for incoming traffic for my gRPC service.

I'd expect this metric to be labeled with the gRPC status code so I can count hits by status code.

Other datadog integrations (such as python flask) already do this with http status code.

mtoffl01 commented 1 month ago

Hello!

Yes, this is a known limitation; http.status_code is a tag on trace metrics but grpc.status.code is not. The grpc spans populate the latter attribute and not the former. See: https://docs.datadoghq.com/tracing/metrics/metrics_namespace/#metric-suffix

A few quick workarounds while the team discusses this further:

Use trace explorer queries to group by grpc status codes on your spans
Create custom span-based metrics with grpc.status.code as a tag: https://docs.datadoghq.com/tracing/trace_pipeline/generate_metrics/