search

iamrajiv / opentelemetry-grpc-gateway-boilerplate

This repository contains a boilerplate project that demonstrates how to set up an OpenTelemetry-enabled gRPC-Gateway REST server. The gRPC server implements a simple SayHello method that returns a greeting message to the client.
MIT License
20 stars 2 forks source link

Passing traceparent header doesnt appear to work #1

Open johnmayock opened 1 year ago

johnmayock commented 1 year ago

Hello. I have followed this guide (and others) but I cannot successfully make a http json request for a grpc endpoint (which has traceparent present in a header of the GET request) to propagate through this OTEL "instrumented" grpc-gateway" following this boilerplate guide.

Use case: An ingress controller is OTEL enabled and generates a trace/span (sent to OTEL collector) upon http request to a service running in k8s. The ingress routes that request to the grpc-gateway (compiled for the service) on port 80.

GET /v1/service-name/api/ HTTP/1.1 traceparent: 00-000000000000000010c04a5dc44899bb-83fda3a85333b7a8-01 User-Agent: Go-http-client/1.1

This works when the request is sent to an auto-instrumented flask application (python) so I know the format of the OTEL w3c context is correct but I am trying to get the grpc-gateway to propagate this so that the span it creates while handling the http request (before it passes toe the grpc service endpoint) is linked as a child of the parent (provided from the inbound http get traceparent).

This guide suggests that setting the otel test map propagator will do that but it doesnt work for me.

I've code inspected the UnaryClientInterceptor here: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5cc3715df180510fa37f11bd6921c7e546dfcf78/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#LL63C61-L63C61

And it seems to not extract the OTEL context but create a new root span and inject that forwards.

This would not propagate the context which is what I am seeing.

Whereas the UnaryStreamInterceptor: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5cc3715df180510fa37f11bd6921c7e546dfcf78/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#L302

does correctly extract the context

ctx = extract(ctx, cfg.Propagators)

    name, attr := spanInfo(info.FullMethod, peerFromCtx(ctx))
    ctx, span := tracer.Start(
        trace.ContextWithRemoteSpanContext(ctx, trace.SpanContextFromContext(ctx)),

Does this boilerplate guide suggest the it propagates a request (with a traceparent header in the http request) through?

The trace example (which uses a stdout trace exporter) shows no propagation, it displays the trace created by the UnaryClientInterceptor but not in the use case where the http request already has a W3C context present.

gsagula commented 1 year ago

I'm seeing similar issue. When calling the grpc endpoint, I see what is expected:

Isito Ingress -> Istio Sidecar -> gRPC-Gateway -> gRPC-Gateway -> Service Span

However, when calling the HTTP endpoint:

gRPC-Gateway -> gRPC-Gateway -> Service Span

Traces don't seen to propagate from Istio to the gRPC-Gateway when using HTTP.

shawn246 commented 12 months ago

because there is no propagation for grpc gateway to extract trace info from http request header then inject trace to grpc request metadata