grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.31k stars 3.37k forks source link

Logs volume stopped showing log level after switching to native OTLP ingestion endpoint #13711

Open jmichalek132 opened 1 month ago

jmichalek132 commented 1 month ago

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. Started Loki (SHA or version) 3.1
  2. Started Java app with the otel java agent (1.25.1) (sending logs to otel collector via OTLP) and otel collector contrib 0.98.0
  3. Switching from using the loki exporter to using the native OTLP ingest endpoint
  4. Query: {service_name="<service_name>"}

Expected behavior I expected the log volume to keep showing the log severity but now it's just showing unknown.

After reading https://grafana.com/docs/loki/latest/send-data/otel/native_otlp_vs_loki_exporter/#formatting-of-logs I assume it's due to with the previous configuration severity would be an indexed label. I wonder if there is a way to restore the behavior with native OTLP log ingestion. To be clear the log level is still present it's just under different name (detected_level) and in the structured metatada instead of the index as far I can tell.

Environment:

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.

image

jmichalek132 commented 1 month ago

Maybe related to https://github.com/grafana/loki/issues/13688