Closed cyrille-leclerc closed 1 year ago
Here is an experiment on the UX introducing otelcol.exporter.loki
, see PROPOSED NEW PIPELINE STEP
blocks:
otelcol.receiver.otlp "default" {
// https://grafana.com/docs/agent/latest/flow/reference/components/otelcol.receiver.otlp/
grpc {
endpoint = "0.0.0.0:4318"
}
output {
metrics = [otelcol.processor.batch.default.input]
logs = [otelcol.processor.batch.default.input]
traces = [otelcol.processor.batch.default.input]
}
}
otelcol.processor.batch "default" {
// https://grafana.com/docs/agent/latest/flow/reference/components/otelcol.processor.batch/
output {
metrics = [otelcol.processor.memory_limiter.default.input]
logs = [otelcol.processor.memory_limiter.default.input]
traces = [otelcol.processor.memory_limiter.default.input]
}
}
otelcol.processor.memory_limiter "default" {
// https://grafana.com/docs/agent/latest/flow/reference/components/otelcol.processor.memory_limiter/
check_interval = "1s"
limit = "50MiB" // alternatively, set `limit_percentage` and `spike_limit_percentage`
output {
metrics = [otelcol.exporter.prometheus.default.input]
// logs = [otelcol.exporter.loki.default.input] // PROPOSED NEW PIPELINE STEP
traces = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
}
}
// PROPOSED NEW PIPELINE STEP
//otelcol.exporter.loki "default" {
// forward_to = [loki.write.default.receiver]
//}
otelcol.exporter.prometheus "default" {
// https://grafana.com/docs/agent/latest/flow/reference/components/otelcol.exporter.prometheus/
forward_to = [prometheus.remote_write.default.receiver]
}
otelcol.exporter.otlp "grafana_cloud_tempo" {
// https://grafana.com/docs/agent/latest/flow/reference/components/otelcol.exporter.otlp/
client {
endpoint = env("TEMPO_ENDPOINT")
auth = otelcol.auth.basic.grafana_cloud_tempo.handler
}
}
loki.write "default" {
// https://grafana.com/docs/agent/latest/flow/reference/components/loki.write/
endpoint {
url = env("LOKI_URL")
http_client_config {
basic_auth {
username = env("LOKI_USERNAME")
password = env("LOKI_PASSWORD")
}
}
}
}
...
See full flow configuration here https://github.com/cyrille-leclerc/my-shopping-cart/blob/d4556f46f93203f0fd055ecaea005809391b2a9d/grafana/agent-config.flow
Thanks for the interest! We're already tracking the inclusion of such OTLP <-> Loki adapters in #2501, apologies if it was hidden away as a separate ticket.
The adapters would work exactly as you envisioned here, glad to see that it makes sense for someone already using Flow!
Thanks for the feature request! As @tpaschalis mentioned, this is a dupe of #2501, #2665, and #2666, so I'm going to close this in favor of those.
Enhancement Request
In order to fully leverage the Grafana Agent OTLP receiver (
otelcol.receiver.otlp
), I want to not only send my metrics to Mimir and traces to Tempo but I also want to send my logs to Loki.To do so, I imagine I would define a Grafana Agent Flow pipeline with an
otelcol.exporter.
for Loki similar to theotelcol.exporter.prometheus
. I would expect consistency of the default setup of the Loki exporter with the Grafana OTLP Gateway.I am aware of the impedance challenges between OTel Logs and Loki, typically on the mapping of the structured field of OTel logs, but
See Also