panic: interface conversion: *v1.ExportTraceServiceRequest is not protoreflect.ProtoMessage: missing method ProtoReflect

[hterik]

Describe the bug Tempo crashes when receiving OLTP telemetry

To Reproduce Steps to reproduce the behavior:

1. Start Tempo via dockerhub image, "grafana/tempo:latest" at time of writing. grafana/tempo@sha256:d6d1497d2b41ec74451dca762698c0321e3729b4bc83245f8b5c6360f481b625 tempo, version (branch: main, revision: 045ede7f) Config:

   distributor:
   receivers:
   otlp:
     protocols:
       grpc:

2. Send some telemetry using Python library, using exact example provided on https://opentelemetry-python.readthedocs.io/en/stable/exporter/otlp/otlp.html

3. Tempo crashes:

   
   panic: interface conversion: *v1.ExportTraceServiceRequest is not protoreflect.ProtoMessage: missing method ProtoReflect

goroutine 423 [running]: github.com/grafana/tempo/pkg/gogocodec.(gogoCodec).Unmarshal(0x1e89f00, {0xc000335800, 0x3c2, 0x3c2}, {0x212c5e0, 0xc000b1a300}) /drone/src/pkg/gogocodec/gogocodec.go:65 +0xb3 google.golang.org/grpc.(Server).processUnaryRPC.func2({0x212c5e0, 0xc000b1a300}) /drone/src/vendor/google.golang.org/grpc/server.go:1263 +0x1f2 go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler({0x1e5f700, 0xc0009be740}, {0x27dc7c0, 0xc000b00a50}, 0xc0004d69c0, 0xc0009e60a0) /drone/src/vendor/go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1/trace_service.pb.go:206 +0x63 google.golang.org/grpc.(Server).processUnaryRPC(0xc0009fa000, {0x2809920, 0xc00017e600}, 0xc000532000, 0xc0009c09c0, 0x3af6660, 0x0) /drone/src/vendor/google.golang.org/grpc/server.go:1286 +0xc8f google.golang.org/grpc.(Server).handleStream(0xc0009fa000, {0x2809920, 0xc00017e600}, 0xc000532000, 0x0) /drone/src/vendor/google.golang.org/grpc/server.go:1609 +0xa2a google.golang.org/grpc.(Server).serveStreams.func1.2() /drone/src/vendor/google.golang.org/grpc/server.go:934 +0x98 created by google.golang.org/grpc.(Server).serveStreams.func1 /drone/src/vendor/google.golang.org/grpc/server.go:932 +0x294

**Expected behavior**
A: Tempo does not crash, even if the data provided is invalid. (I don't know enough about the opentelemetry-oython nor the wire protocol to tell if it is or not)
B: Tempo accepts the telemetry provided, assuming it was valid.

[mdisibio]

Hi thanks for reporting this. We are aware and working on a fix. The main branch has this issue after merging #1186.

[hterik]

Thanks, i should have been using latest tagged release instead 1.2.1. It doesn't have this problem :+1:

[annanay25]

@hterik, thanks for reporting this. I believe this should be fixed now. We've also added some e2e tests to ensure this doesn't break again. Feel free to reopen if you continue to see this issue!

[sergeyshaykhullin]

I am on Tempo 1.3 and getting

{"caller":"main.go:189","level":"info","msg":"initialising OpenTracing tracer","ts":"2022-01-31T17:10:04.572391548Z"} {"caller":"main.go:108","level":"info","msg":"Starting Tempo","ts":"2022-01-31T17:10:05.483348333Z","version":"(version=, branch=HEAD, revision=be6476d9a)"} {"caller":"server.go:260","grpc":"[::]:9095","http":"[::]:3100","level":"info","msg":"server listening on addresses","ts":"2022-01-31T17:10:05.785617114Z"} ts=2022-01-31T17:10:06Z level=info msg="OTel Shim Logger Initialized" component=tempo {"caller":"module_service.go:64","level":"info","module":"server","msg":"initialising","ts":"2022-01-31T17:10:06.077948536Z"} {"caller":"module_service.go:64","level":"info","module":"overrides","msg":"initialising","ts":"2022-01-31T17:10:06.078329158Z"} {"caller":"module_service.go:64","level":"info","module":"memberlist-kv","msg":"initialising","ts":"2022-01-31T17:10:06.078404801Z"} {"caller":"module_service.go:64","level":"info","module":"ring","msg":"initialising","ts":"2022-01-31T17:10:06.078528142Z"} panic: interface conversion: *etcdserverpb.RangeRequest is not protoreflect.ProtoMessage: missing method ProtoReflect goroutine 283 [running]: github.com/grafana/tempo/pkg/gogocodec.(*gogoCodec).Marshal(0x3b6d418, {0x21aa8c0, 0xc000205050}) /drone/src/pkg/gogocodec/gogocodec.go:55 +0x90 google.golang.org/grpc.encode({0x7fcdf6cb43f8, 0x3b6d418}, {0x21aa8c0, 0xc000205050}) /drone/src/vendor/google.golang.org/grpc/rpc_util.go:593 +0x44 google.golang.org/grpc.prepareMsg({0x21aa8c0, 0xc000205050}, {0x7fcdf6cb43f8, 0x3b6d418}, {0x0, 0x0}, {0x0, 0x0}) /drone/src/vendor/google.golang.org/grpc/stream.go:1590 +0xd2 google.golang.org/grpc.(*clientStream).SendMsg(0xc0008ea240, {0x21aa8c0, 0xc000205050}) /drone/src/vendor/google.golang.org/grpc/stream.go:771 +0x176 google.golang.org/grpc.invoke({0x27e1f20, 0xc000438c00}, {0x223a450, 0x0}, {0x21aa8c0, 0xc000205050}, {0x2166200, 0xc000099cc0}, 0x0, {0xc000534a40, ...}) /drone/src/vendor/google.golang.org/grpc/call.go:70 +0xa8 go.etcd.io/etcd/client/v3.(*Client).unaryClientInterceptor.func1({0x27e1f20, 0xc000438ba0}, {0x223a450, 0x16}, {0x21aa8c0, 0xc000205050}, {0x2166200, 0xc000099cc0}, 0xc0008c4000, 0x23932c0, ...) /drone/src/vendor/go.etcd.io/etcd/client/v3/retry_interceptor.go:58 +0x42a google.golang.org/grpc.(*ClientConn).Invoke(0x60, {0x27e1f20, 0xc000438ba0}, {0x223a450, 0x6}, {0x21aa8c0, 0xc000205050}, {0x2166200, 0xc000099cc0}, {0xc000477680, ...}) /drone/src/vendor/google.golang.org/grpc/call.go:35 +0x21f go.etcd.io/etcd/api/v3/etcdserverpb.(*kVClient).Range(0xc00012e460, {0x27e1f20, 0xc000438ba0}, 0xc000242000, {0xc000477680, 0x4, 0x6}) /drone/src/vendor/go.etcd.io/etcd/api/v3/etcdserverpb/rpc.pb.go:6460 +0xc9 go.etcd.io/etcd/client/v3.(*retryKVClient).Range(0xc000831320, {0x27e1f20, 0xc000438ba0}, 0x0, {0x3b34640, 0x3, 0x3}) /drone/src/vendor/go.etcd.io/etcd/client/v3/retry.go:105 +0x130 go.etcd.io/etcd/client/v3.(*kv).Do(0xc0002f71a0, {_, _}, {0x1, {0xc0003655e0, 0xa, 0x10}, {0x0, 0x0, 0x0}, ...}) /drone/src/vendor/go.etcd.io/etcd/client/v3/kv.go:149 +0x1e8

[annanay25]

Thanks for reporting this @sergeyshaykhullin, @mapno has a PR up to fix this, we'll try to release a patch soon with the fix included.

[kostix]

@annanay25, can you please provide a link to that PR? I've failed to find it in the issue tracker.