ZipkinSpanExporter - Failed to export spans

pkemp-bh commented 2 years ago

What did you try to do? Integrate an app with splunk-otel-javaagent-all.jar v1.13.0 to send spans to SignalFX

What happened? Some (not all) spans are being sent successfully, but we are seeing an exception every few seconds in our logs:

[otel.javaagent 2022-07-08 15:09:19:919 +0000] [OkHttp http://10.132.18.189:9080/...] WARN io.opentelemetry.exporter.zipkin.ZipkinSpanExporter - Failed to export spans
java.lang.RuntimeException: response for null failed: error handling gzip compressed request EOF
    at zipkin2.reporter.okhttp3.HttpCall.parseResponse(HttpCall.java:93)
    at zipkin2.reporter.okhttp3.HttpCall$V2CallbackAdapter.onResponse(HttpCall.java:68)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

What did you expect to happen? All spans to be sent and no exceptions in the logs.
Have you found any workaround? No
How impactful is the issue? It is preventing us from properly monitoring an app, so highly impactful.
How can we reproduce the issue? We do not have a minimal reproduction at this time; we have only observed this happening in one of our Spring Boot apps.
What OS, JVM, libraries, frameworks, application servers are you using? Kubernetes (docker), "OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)", Spring Boot v2.6.3

What configuration options were used? No special java options. Environment vars for the app:

        - name: OTEL_EXPORTER_ZIPKIN_ENDPOINT
          value: http://$(SIGNALFX_AGENT_HOST):9080/v1/trace
        - name: OTEL_TRACES_EXPORTER
          value: zipkin
        - name: OTEL_SERVICE_NAME
          value: app-dev

Snippet of k8s config for the collector:

image:
      otelcol:
        repository: quay.io/signalfx/splunk-otel-collector
        tag: 0.35.0 #version
        pullPolicy: IfNotPresent

    receivers:
      sapm:
        endpoint: 0.0.0.0:7276
      smartagent/docker-container-stats:
        dockerURL: unix:///var/run/docker.sock
        excludedImages:
        - '*pause-amd64*'
        - k8s.gcr.io/pause*
        labelsToDimensions:
          io.kubernetes.container.name: container_spec_name
          io.kubernetes.pod.name: kubernetes_pod_name
          io.kubernetes.pod.namespace: kubernetes_namespace
          io.kubernetes.pod.uid: kubernetes_pod_uid
        type: docker-container-stats
      smartagent/kubelet-metrics:
        kubeletAPI:
          authType: serviceAccount
          url: https://localhost:10250
        type: kubelet-metrics
      smartagent/signalfx-forwarder:
        defaultSpanTags:
          environment: nonprod 
        listenAddress: 0.0.0.0:9080
        type: signalfx-forwarder

    exporters:
      sapm:
        access_token: ...
        access_token_passthrough: true
        endpoint: https://ingest.us1.signalfx.com/v2/trace
        max_connections: 100
        num_workers: 8

    processors:
      batch:
      batch/2:
        #send_batch_size: 10000
        timeout: 10s
        send_batch_max_size: 10000 #(default = 0): The upper limit of the batch size. 0 means no upper limit of the batch size. This property ensures that larger batches are split into smaller units. It must be greater or equal to send_batch_size.

Debug logs: otel_logs.txt

mateuszrzeszutek commented 2 years ago

Hey @pkemp-bh ,

Have you tried configuring the collector to use a zipkin receiver instead of the signalfx-forwarder? Also, the collector image you're running is 10 months old; have you tried using a more recent version? If so, does the problem still persist?

If it is possible for you, I'd also strongly recommend using OTLP as the trace protocol - Zipkin as a protocol has lots of downsides, e.g. you're losing resource data, span links, and the span events are severely limited (no attributes).

breedx-splk commented 2 years ago

Hi @pkemp-bh. Have you had a chance to look at the question asked on Jul 11? We will close this issue as stale if we don't get a response. Thanks.

mateuszrzeszutek commented 1 year ago

Hey @pkemp-bh , Please reopen this issue or create a new one if you're still experiencing this problem.

signalfx / splunk-otel-java

ZipkinSpanExporter - Failed to export spans #840