canonical / loki-k8s-operator

https://charmhub.io/loki-k8s
Apache License 2.0
10 stars 16 forks source link

add pipeline_stage to remove `filename` label #455

Closed Abuelodelanada closed 2 months ago

Abuelodelanada commented 2 months ago

Issue

This PR goes in tandem with https://github.com/canonical/grafana-agent-operator/pull/180

Solution

We add pipeline_stages to promtail configuration in order to remove filename label an add it as structured_metadata

Testing Instructions

Deploy this bundle (update lib in flog and pack loki)

bundle: kubernetes
applications:
  flog:
    charm: flog-k8s
    channel: latest/edge
    revision: 8
    resources:
      workload-image: 2
    scale: 1
    constraints: arch=amd64
  grafana:
    charm: grafana-k8s
    channel: latest/edge
    revision: 118
    resources:
      grafana-image: 70
      litestream-image: 45
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  loki:
    charm: local:loki-k8s-0
    scale: 1
    constraints: arch=amd64
    storage:
      active-index-directory: kubernetes,1,1024M
      loki-chunks: kubernetes,1,1024M
    trust: true
  traefik:
    charm: traefik-k8s
    channel: latest/edge
    revision: 207
    resources:
      traefik-image: 160
    scale: 1
    constraints: arch=amd64
    storage:
      configurations: kubernetes,1,1024M
    trust: true
relations:
- - traefik:ingress-per-unit
  - loki:ingress
- - grafana:ingress
  - traefik:traefik-route
- - grafana:grafana-source
  - loki:grafana-source
- - flog:log-proxy
  - loki:logging

Check in Grafana that filename is not a regular label but part of structured_metadata

image

image

Abuelodelanada commented 2 months ago

Since we have learn something that was not documented in Grafana docs, which is that grafana-agent is not able to forward structured_metadata coming from promtail (or another grafana-agent) to loki

promtail --> grafana-agent --> loki

We are closing this PR because the pipeline is not reliable. An administrator may wrongly think that because promtail sends structured_metadata it will reach loki when it is not always true.