Jaeger gRPC receiver doesn't work properly

[fpaparoni]

Describe the bug Jaeger gRPC receiver doesn't work properly

Steps to reproduce I'm using the collector as agent to Jaeger HTTP Thrift endpoint. I'm connecting from opentelemetry client sending traces to 14250 gRPC port and I setup export to 14268 Jaeger thrift endoint

What did you expect to see? Traces in Jaeger

What did you see instead? From the opentelemetry client I receive this error :

test-app_1  | [opentelemetry.auto.trace 2021-02-26 13:36:41:390 +0000] [grpc-default-executor-0] WARN io.opentelemetry.exporter.jaeger.JaegerGrpcSpanExporter - Failed to export spans
test-app_1  | io.grpc.StatusRuntimeException: UNKNOWN: HTTP 404 "Not Found"
test-app_1  |   at io.grpc.Status.asRuntimeException(Status.java:534)
test-app_1  |   at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
test-app_1  |   at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)
test-app_1  |   at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)
test-app_1  |   at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)
test-app_1  |   at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
test-app_1  |   at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
test-app_1  |   at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
test-app_1  |   at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
test-app_1  |   at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
test-app_1  |   at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
test-app_1  |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
test-app_1  |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
test-app_1  |   at java.lang.Thread.run(Thread.java:748)

Using Collector core distribution with the same receiver I don't have this problem, client connects without problems, but unfortunately I don't have the thrift exporter that I need

What version did you use? Collector-contrib Version: v0.21.0 Collector Version: v0.21.0 opentelemetry-javaagent: v.0.16.1

What config did you use? Collector config:

receivers:
  jaeger:
    protocols:
      grpc:

exporters:
 jaeger_thrift:
   url: "http://jaeger:14268"
   timeout: 2s

extensions:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679
  health_check:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [jaeger]
      exporters: [jaeger_thrift]

Environment OS: Ubuntu 20.04

[bogdandrutu]

/cc @jkwatson @iNikem I think this is similar with the problem discussed on Slack

[bogdandrutu]

/cc @jpkrohling we were talking about removing this and you said that this exporter is not used :)

[iNikem]

@bogdandrutu what Slack discussion do you have in mind? Can you give a link?

[jpkrohling]

/cc @jpkrohling we were talking about removing this and you said that this exporter is not used :)

No no no, we talked about removing the Jaeger Thrift HTTP Exporter :-) The gRPC receiver is the most used way to consume data in Jaeger, being the preferred channel for the agent -> collector for quite a long time now.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerthrifthttpexporter

Edit: sorry, just saw the configuration used in this report and you are right. In any case, @fpaparoni should be using the gRPC exporter instead. Any specific reasons you are using the Thrift HTTP exporter? Also, make sure to specify the /api/traces path as part of the exporter's URL, as per the docs: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerthrifthttpexporter#configuration

[fpaparoni]

Any specific reasons you are using the Thrift HTTP exporter?

Unfortunately I cannot use the gRPC exporter (project problems...) so the only way to connect to our Jaeger istance is to use the Thrift exporter

Also, make sure to specify the /api/traces path as part of the exporter's URL, as per the docs: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerthrifthttpexporter#configuration

The problem isn't related to the exporter but to the receiver, because we have the connection problem with our client application that is trying to connect to the collector

[jpkrohling]

This sounds strange, but I'll take a look in the next few days.

[jpkrohling]

I'm able to reproduce this and I'm working on it.

[jpkrohling]

I got to reproduce the error message you got but turns out, it's indeed what I mentioned before:

Also, make sure to specify the /api/traces path as part of the exporter's URL, as per the docs: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerthrifthttpexporter#configuration

Changing the url in your example to include /api/traces made it work for me. Here's my YAML:

receivers:
  jaeger:
    protocols:
      grpc:

exporters:
 jaeger_thrift:
   url: "http://localhost:14268/api/traces"
   timeout: 2s

extensions:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679
  health_check:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [jaeger]
      exporters: [jaeger_thrift]

I used the opentelemetry-generate-span-java example, changing from the OTLP exporter to Jaeger. Jaeger was then started with: $ podman run --rm -p 14268:14268 -p 16686:16686 jaegertracing/all-in-one:1.22.0 --log-level=debug and the otelcol was executed from the sources, with $ go run ./cmd/otelcontribcol --config /tmp/otelcol.yaml

[jpkrohling]

Feel free to reopen if adding the path to the URL does not solve your problem.