Allow out-of-order exemplars

[justinbwood]

Describe the bug

Mimir rejects out-of-order exemplars, even when out-of-order samples are configured.

To Reproduce

Steps to reproduce the behavior:

1. Deploy and configure Mimir with out-of order samples enabled.
2. Deploy Grafana Agent to accept OTLP and write to Mimir
3. Configure a source of OTLP datapoints with exemplars (For example Micrometer with OTel Java Agent)

Expected behavior

Mimir allows out-of-order exemplars, or can be configured to allow it.

Environment

• Infrastructure: Kubernetes
• Deployment tool: Helm

Additional Context

Partial Helm values for Mimir:

runtimeConfig:
  overrides:
    anonymous:
      out_of_order_time_window: 5m

Partial Grafana Agent config:

otelcol.receiver.otlp "traces" {
  http {}

  output {
    metrics = [otelcol.connector.spanmetrics.red.input]
  }
}

otelcol.connector.spanmetrics "red" {
  histogram {
    explicit {}
  }

  exemplars {
    enabled = true
  }

  output {
    metrics = [otelcol.exporter.prometheus.mimir.input]
  }
}

otelcol.exporter.prometheus "mimir" {
  forward_to = [prometheus.remote_write.mimir.receiver]
}

Resulting log message from Agent:

ts=2024-03-28T21:22:49.191638465Z level=error msg="non-recoverable error" component=prometheus.remote_write.mimir subcomponent=rw remote_name=546f14 url=https://mimir.example.com/api/v1/push count=1943 exemplarCount=57 err="server returned HTTP status 400 Bad Request: failed pushing to ingester: user=anonymous: err: out of order exemplar. timestamp=2024-03-28T21:22:09.783Z, 
  series={__name__=\"duration_milliseconds_bucket\", env=\"dev\", http_method=\"GET\", http_status_code=\"200\", job=\"job-xxx\", le=\"2\", namespace=\"ns-xxx\", service=\"svc-xxx\", span_kind=\"SPAN_KIND_SERVER\", span_name=\"GET /xxx/api/v1/**\", status_code=\"STATUS_CODE_UNSET\"}, exemplar={trace_id=\"5f23d20e507fb4c177566f9782097345\", span_id=\"d3ac98b035724486\"}"

[56quarters]

Corresponding upstream issue: https://github.com/prometheus/prometheus/issues/13577

[justinbwood]

Thanks for the context!

If there are any suggestions to reduce the likelihood that I'm creating out-of-order exemplars, I'm all ears. Roughly speaking, my OTLP ingestion pipeline goes:

1. OTLP/HTTP receiver
2. memory limiter
3. k8s attributes from pod association
4. export traces to otelcol loadbalancer
5. receiver for loadbalancer
6. spanmetrics processor
7. transform k8s resource attributes like pod name to datapoint attributes
8. otelcol prometheus exporter
9. remote write to mimir