uploadTrace by grpc has something wrong

[bai-ming]

Description

I send the trace to tempo, the code works without any error, but I can't find the trace in tempo.

Environment

• Go Version: 1.19
• opentelemetry-go version: v1.14.0

Steps To Reproduce

I set tp with code below

func InitTracerProvider() error {
    exp, err := newExporter(utils.OsSignalHandler.SubContext())
    // exp, err := newStdOutExporter()
    if err != nil {
        return errors.Wrap(err, "")
    }
    tp = sdk.NewTracerProvider(
        sdk.WithBatcher(exp),
        sdk.WithIDGenerator(newTempIDGenerator()),
        sdk.WithResource(newResource()),
    )
    otel.SetTracerProvider(tp)
    return nil
}

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
        tempo := "localhost:4317"
        exp, err := otlptracegrpc.New(ctx, otlptracegrpc.WithInsecure(), otlptracegrpc.WithEndpoint(tempo))
        return exp, errors.Wrap(err, "exporter init failed")
}

when I use v1.14.0 It works well, It sends message without error and I can find the trace in tempo which I send. But when I use v1.21.0, It still sends message without any error but I can't find it in tempo. And tempo dose not have any error log.

the difference about two version mod is the picture below

So perhaps It has something about the package?

[pellared]

From the description I do not see any proof that something is wrong with the exporter. Also the reproduction steps are not precise (not working code, missing info about Grafana Tempo version and its configuration). Please provide Minimal, Reproducible Example.

However, I personally suggest creating an issue in https://github.com/grafana/tempo. If there is a bug in our exporter, their maintainers should be able to locate the problem.

[Tsuribori]

I have run into the same issue when going from v1.16.0 library versions to v1.19.0, and the issue persists in v1.21.0. It doesn't seem to be a Tempo issue as it affects also Jaeger. I tested with Tempo 2.1.1, Tempo 2.3.1 and Jager jaegertracing/all-in-one:1.52 and the issue is present in all of them, they can't show traces correctly after the upgrade.

With v1.16.0 in Jaeger:

With v1.21.0 in Jaeger:

Something funky seem to be going on...

The tracing setup code used is as following (with the grpc path):

func Initialize() error {
    if !config.GetTracingEnabled() {
        return nil
    }

    insecure := config.GetTracingInsecureTransport()

    var tpo trace.TracerProviderOption
    switch config.GetTracingTransport() {
    case "grpc":
        opts := []otlptracegrpc.Option{
            otlptracegrpc.WithEndpoint(config.GetTracingEndpoint()),
        }
        if insecure {
            opts = append(opts, otlptracegrpc.WithInsecure())
        }
        exp, err := otlptracegrpc.New(context.Background(), opts...)
        if err != nil {
            return fmt.Errorf("building tracing exporter: %w", err)
        }
        tpo = trace.WithBatcher(exp)
    case "http":
        opts := []otlptracehttp.Option{
            otlptracehttp.WithEndpoint(config.GetTracingEndpoint()),
        }
        if insecure {
            opts = append(opts, otlptracehttp.WithInsecure())
        }
        exp, err := otlptracehttp.New(context.Background(), opts...)
        if err != nil {
            return fmt.Errorf("building tracing exporter: %w", err)
        }
        tpo = trace.WithBatcher(exp)
    default:
        return fmt.Errorf("invalid tracing transport: %s", config.GetTracingTransport())
    }
    r, _ := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("GoToSocial"),
        ),
    )

    tp := trace.NewTracerProvider(
        tpo,
        trace.WithResource(r),
    )
    otel.SetTracerProvider(tp)
    propagator := propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    )
    otel.SetTextMapPropagator(propagator)
    log.Hook(func(ctx context.Context, kvs []kv.Field) []kv.Field {
        span := oteltrace.SpanFromContext(ctx)
        if span != nil && span.SpanContext().HasTraceID() {
            return append(kvs, kv.Field{K: "traceID", V: span.SpanContext().TraceID().String()})
        }
        return kvs
    })
    return nil
}

OpenTelemetry libraries used:

    go.opentelemetry.io/otel v1.21.0
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.21.0
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.21.0
    go.opentelemetry.io/otel/exporters/prometheus v0.44.0
    go.opentelemetry.io/otel/sdk v1.21.0
    go.opentelemetry.io/otel/sdk/metric v1.21.0
    go.opentelemetry.io/otel/trace v1.21.0
    github.com/uptrace/opentelemetry-go-extra/otelsql v0.2.2 // indirect
    go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.21.0 // indirect
    go.opentelemetry.io/otel/metric v1.21.0 // indirect
    go.opentelemetry.io/proto/otlp v1.0.0 // indirect

Go version used is 1.21

[pellared]

You miss error handling for r, _ := resource.Merge

My guess is that you have forgotten to bump the semconv in your import path to semconv "go.opentelemetry.io/otel/semconv/v1.21.0" when bumping OTel Go.

We are perfectly aware that this issue is annoying. Already tracked under: https://github.com/open-telemetry/opentelemetry-go/issues/2341

[Tsuribori]

You miss error handling for r, _ := resource.Merge

My guess is that you have forgotten to bump the semconv in your import path to semconv "go.opentelemetry.io/otel/semconv/v1.21.0" when bumping OTel Go.

We are perfectly aware that this issue is annoying. Already tracked under: #2341

Ah! That was the problem, thank you.