[Tempo-distributed] - Unclear override configuation for tempo-distributed

[Nierhoff]

Describe the bug I am trying to increase the rate limits in tempo, deployed using helm and the chart tempo-distributed and has run into issues understanding the documentation.

To Reproduce Steps to reproduce the behavior: Deployed using command: helm install tempo-distributed grafana/tempo-distributed --namespace tempo --create-namespace --values distributed.yaml

the value file "distributed.yaml" content:

storage:
  trace:
    backend: s3
    s3:
      bucket: "tempo-traces"
      endpoint: minio.minio.svc.cluster.local:80
      access_key: tempo
      secret_key: REPLACE
      forcepathstyle: true
      insecure: true

global_overrides:
  per_tenant_override_config: /runtime-config/overrides.yaml
overrides: |-
  overrides:
    defaults:
      ingestion:
        burst_size_bytes: 20000000 # 20MB
        rate_limit_bytes: 20000000 # 20MB
        max_traces_per_user: 10000
        ingestion_burst_size_bytes: 20000000 # 20MB
        ingestion_rate_limit_bytes: 20000000 # 20MB
      global:
        max_bytes_per_trace: 20000000 # 20MB
    "*":
      ingestion:
        burst_size_bytes: 20000000 # 20MB
        rate_limit_bytes: 20000000 # 20MB
        max_traces_per_user: 10000
        ingestion_burst_size_bytes: 20000000 # 20MB
        ingestion_rate_limit_bytes: 20000000 # 20MB
      global:
        max_bytes_per_trace: 20000000 # 20MB

ingester:
   replicas: 1

traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true

I have seen multiple errors: the current is: "field ingestion not found in type overrides.LegacyOverrides"

The question is

1. How should "global_overrides" be used ?
2. what is the correct format for "overrides:" ?

I have been trying to use this as a guide https://grafana.com/docs/tempo/latest/configuration/#runtime-overrides

Expected behavior I am sure this is me not fully understanding the documentation. A single example for a value file for my version (1.9.9) would be much appreciated, where typical rate limit configuation is done. It might be related to https://github.com/grafana/helm-charts/issues/2802

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm
• helm Chart version: 1.9.9
• Tempo version: 2.4.1

Link to issue in tempo repo: https://github.com/grafana/helm-charts/issues/3171

[benmathews]

I came here to file the same ticket. By examining the tempo and chart source, I was able to puzzle out the correct configuration. (And it has nothing to do with what is documented.)

Put something like this in your overrides.yaml file.

tempo:
  structuredConfig:
    overrides:
      defaults:
        ingestion:
          rate_limit_bytes: 40000000
          burst_size_bytes: 50000000
          max_traces_per_user: 30000
        global:
          max_bytes_per_trace: 8000000

[vterdunov]

I came here to file the same ticket. By examining the tempo and chart source, I was able to puzzle out the correct configuration. (And it has nothing to do with what is documented.)

Put something like this in your overrides.yaml file.

tempo:
  structuredConfig:
    overrides:
      defaults:
        ingestion:
          rate_limit_bytes: 40000000
          burst_size_bytes: 50000000
          max_traces_per_user: 30000
        global:
          max_bytes_per_trace: 8000000

This fails for me with

failed parsing config: failed to parse configFile /conf/tempo.yaml: yaml: unmarshal errors:
  line 76: field defaults not found in type overrides.legacyConfig

Chart: 1.9.11 App Version: 2.4.2

[benmathews]

Chart

I'm using chart 1.9.10, but don't see anything in the diff that would matter. :shrug:

[pantuza]

If you want to workaround this Helm Chart issue, just use as follows:

global_overrides:
  defaults:
    ingestion:
      rate_limit_bytes: 32000000 # 32MB
      burst_size_bytes: 48000000 # 48MB
      max_traces_per_user: 50000

Here I explain a little bit further why the issue is happening.