"No data for selected query" in Explore Tempo menu

[eunhwa-park]

Hi, I’m trying to use the Explore Tempo plugin. I've connected the Tempo data source, and traces are visible, but in the RED metrics or other sections, it shows "No data for selected query." There are no errors in the logs of each pod either.

• Tempo App version : 2.6.0
• Tempo Chart : tempo-distributed-1.18.4

What could be misconfigured? Below is the values.yaml file.

traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true
tempo:
  structuredConfig:
    ingester:
      lifecycler:
        ring:
          replication_factor: 3
      max_block_bytes: 104857600
      max_block_duration: 10m
      complete_block_timeout: 15m

storage:
  trace:
    backend: s3
    s3:
      region: ap-northeast-2
      bucket: owl-tempo
      endpoint: s3.ap-northeast-2.amazonaws.com
      insecure: true
    search:
      cache_control:
        footer: true
    pool:
      max_workers: 400
      queue_depth: 20000
    wal:
      path: /var/tempo/wal

distributor:
  replicas: 2
  config:
    log_received_spans:
      enabled: true

ingester:
  replicas: 3
  persistence:
    enabled: true
    size: 10Gi

serviceAccount:
  create: true
  name: "tempo"
  annotations:
    "eks.amazonaws.com/role-arn": "arn:aws:iam::xxxxx:role/xxxx-tempo-irsa"
  automountServiceAccountToken: true

global_overrides:
  metrics_generator_processors:
    - service-graphs
    - span-metrics
    - local-blocks
metricsGenerator:
  enabled: true
  config:
    processor:
      local_blocks:
        flush_to_storage: true

Thank you!

[joe-elliott]

If you run a basic traceql metrics query in normal explore do you get a result? something like:

{} | rate()

[eunhwa-park]

No,, when I query only {}, I get a result, but when I add{} | rate(), no result is returned. Is there any additional configuration needed for the rate function?

[joe-elliott]

Here is a doc about configuring tempo for

https://grafana.com/docs/tempo/latest/operations/traceql-metrics/#before-you-begin

and i think a few details for 2.6 are missing that can be found in the release notes (cc @knylander-grafana)

https://grafana.com/docs/tempo/latest/release-notes/v2-6/#operational-change-for-traceql-metrics

unfortunately, i don't know much about helm. if you provide the generated tempo config, i can probably give better advice.

[eunhwa-park]

I referenced the shared page and added it to metricsGenerator, but TraceQL still doesn't work properly. There is no /var/tempo/traces path in the tempo-metrics-generator service pod. Could this be due to incorrect configuration?

$ tree /var/tempo/
/var/tempo/
└── wal
    └── blocks

I am sharing the Tempo configuration. If there are any missing parts, please let me know. Thanks a lot!

traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true
tempo:
  structuredConfig:
    ingester:
      lifecycler:
        ring:
          replication_factor: 3
      max_block_bytes: 104857600
      max_block_duration: 10m
      complete_block_timeout: 15m

storage:
  trace:
    backend: s3
    s3:
      region: ap-northeast-2
      bucket: xxx-tempo
      endpoint: s3.ap-northeast-2.amazonaws.com
      insecure: true
    search:
      cache_control:
        footer: true
    pool:
      max_workers: 400
      queue_depth: 20000
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/traces

distributor:
  replicas: 2
  config:
    log_received_spans:
      enabled: true

ingester:
  replicas: 3
  persistence:
    enabled: true
    size: 10Gi

serviceAccount:
  create: true
  name: "tempo"
  annotations:
    "eks.amazonaws.com/role-arn": "arn:aws:iam::xxxx:role/xxx-tempo-irsa"
  automountServiceAccountToken: true

global_overrides:
  metrics_generator_processors:
    - service-graphs
    - span-metrics
    - local-blocks
metricsGenerator:
  enabled: true
  config:
    processor:
      local_blocks:
        flush_to_storage: true
        filter_server_spans: false
    storage:
      path: /var/tempo/wal
    traces_storage:
      path: /var/tempo/traces

[knylander-grafana]

Here is a doc about configuring tempo for

https://grafana.com/docs/tempo/latest/operations/traceql-metrics/#before-you-begin

and i think a few details for 2.6 are missing that can be found in the release notes (cc @knylander-grafana)

I've updated the configuration docs to add the 2.6 info here.

If you find anything else missing in the docs, please let me know and we'll get them updated.

[joe-elliott]

I think you're sharing the values.yaml file from the helm chart, but I'm not very familiar with that. If you could share the actual rendered Tempo configmap I would probably be able to find the misconfiguration.

[eunhwa-park]

Thanks for your support. Here is my tempo-config configmap yaml.

tempo-config-cm.txt