Overrides Configuration Is Not Working

[randrewy]

Describe the bug Docs states that starting from version 2.3 there is a new overrides block.

To Reproduce Steps to reproduce the behavior:

1. Start Tempo with the following overrides section in config:

   overrides:
   defaults:
   metrics_generator:
     processors: [service-graphs, span-metrics]

2. get the error: line 38: field defaults not found in type overrides.legacyConfig

Expected behavior Server starts.

Environment:

• Infrastructure: local docker
• Deployment tool: docker run

Additional Context image "grafana/tempo:2.5.0"

At the same time another line from docs shows old format: "A snippet of a config.yaml file showing how the overrides section is here.". And with that legacy configuration everything works fine.

[mapno]

Hi! Do you have per-tenant overrides? You need to use the same format in both overrides configs.

[randrewy]

Hello! I have multi-tenancy disabled in the same config file

[ftong2020]

indeed, I am encountering this issue, too. It is quite confusing. Documentation is no where to be found how tempo tells which configuration is "new" and which is "legacy".

[AlexDCraig]

I get this too using the tempo-distributed helm chart and:

global_overrides:
  defaults:
    metrics_generator:
      processors:
        - span-metrics
      processor:
        span_metrics:
          histogram_buckets: [null]
          dimensions:
            - kubernetes_cluster
            - kubernetes.cluster
            - http.status_code

[mapno]

Documentation is no where to be found how tempo tells which configuration is "new" and which is "legacy".

That's fair. We probably didn't add the best documentation of the change. As a rule of thumb: long, unindented params are legacy (eg. metrics_generator_forwarder_queue_size), indented params are new (eg. metrics_generator.forwarder.queue_size).

With the new config, a new key defaults was also added after the overrides key, to be more explicit of the contents of the config block.

Finally, there is a command in the tempo-cli that allows you to convert legacy config to new overrides config.

[knylander-grafana]

Thank you for the feedback on the docs. I've created a doc issue to make sure that we capture these changes and address the issues you've had.

https://github.com/grafana/tempo/issues/3820