grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
3.93k stars 510 forks source link

Problems with integration with opentelemetry collector #4052

Closed Ventura94 closed 3 weeks ago

Ventura94 commented 3 weeks ago

Opentelemetry logs:

otelcol-contrib[33599]: 2024-09-04T18:05:49.789Z        error        exporterhelper/queue_sender.go:92        Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", "dropped_items": 1}

I have the tempo services and the opentelemetry collector on the same local server with these configurations:

Opentelemetry:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:55680
      http:
        endpoint: 0.0.0.0:55681
exporters:
  prometheus:
    endpoint: "0.0.0.0:3300"

  logging:

  otlp:
    endpoint: "localhost:9095"
    tls:
      insecure: true

  loki:
    endpoint: http://localhost:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp, logging]

    metrics:
      receivers: [otlp]
      exporters: [prometheus, logging]

    logs:
      receivers: [otlp]
      exporters: [loki, logging]

Tempo:


stream_over_http_enabled: true
server:
  http_listen_port: 3200
  grpc_listen_port: 9095

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 5s

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    otlp:
      protocols:
        http:
        grpc:

ingester:
  max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
  compaction:
    block_retention: 1h                # overall Tempo trace retention. set for demo purposes

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: /var/tempo/wal             # where to store the wal locally
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks] # enables metrics generator
      generate_native_histograms: both

I don't understand what I'm doing wrong

joe-elliott commented 3 weeks ago

Closing b/c you also opened a discussion here: https://github.com/grafana/tempo/discussions/4053