gRPC application integration using java agent

[satyav-equinix]

Describe the bug We have a GRPC application and we are trying to use open telemetry for distributed tracing. The setup we are using on a high level is microservices export spans to otel-collector(running as a separate workload) which exports the data to jaeger.

The problem is that we are seeing the below message in the logs and also we noticed spans are being received by otel-collector.

[otel.javaagent 2022-03-08 15:29:08:528 -0800] [OkHttp http://localhost:4318/...] WARN io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter - Failed to export spans. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found

VM args and Env variables::

-javaagent:/opentelemetry-javaagent.jar

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318 OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf

What version are you using? opentelemetry-java-agent:1.7.2

Environment openjdk:8u275

[anuraaga]

Hi @satyav-equinix - does it work if you set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318? Also it would be good to update to the latest java agent to make sure not hitting an old issue.

[satyav-equinix]

Thanks @anuraaga, let me try with OTEL_EXPORTER_OTLP_ENDPOINT and the latest version as well and update here.

[satyav-equinix]

@anuraaga I tried with env var OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 and later agent jar. still same error.

[anuraaga]

@satyav-equinix Can you post / confirm the configuration of otel-collector? Does it enable the http protocol? 404 means it is not being able to find the endpoint so this could be an issue with the collector not enabling it.

receivers:
  otlp:
    protocols:
      grpc:
      http:

[satyav-equinix]

@anuraaga here is the otel-config info from otel-collector.

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
  memory_limiter:
    # 80% of maximum memory up to 2G
    limit_mib: 1500
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s
extensions:
  zpages: {}
  memory_ballast:
    # Memory Ballast size should be max 1/3 to 1/2 of memory.
    size_mib: 683
exporters:
  jaeger:
    endpoint: "0.0.0.0:14250" # Replace with a real endpoint.
    tls:
      insecure: true
service:
  extensions: [zpages, memory_ballast]
  pipelines:
    traces/1:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [jaeger]

[anuraaga]

Things generally look good - by the way why do you set to http/protobuf protocol instead of using all the defaults, which would use gRPC protocol to localhost:4317? Do you have a HTTP proxy in front of the collector? Can you confirm /v1/traces is accessible through it if so?

[satyav-equinix]

@anuraaga I don't have an exact reason why we choose http/protobuf protocol instead of defaults(since this was initially implemented by someone else). I can try the default gRPC protocol with 4317(will check and update on this). We don't have any HTTP proxy in front of otel-collector. But tried the connectivity, here is the result(in SS). is there any specific format I should try in the request body? Also, along the same lines, is the otel java agent looking for any specific log pattern? The log pattern which we are using is <pattern>%-5p | %t | %d | %c{5} | %m%n</pattern>

[trask]

@satyav-equinix are you still having this issue? is there any more details you can share that might give us more clues?

[mateuszrzeszutek]

Ping @satyav-equinix

[github-actions[bot]]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.