grafana / helm-charts

Apache License 2.0
1.67k stars 2.28k forks source link

helm install loki-stack -f custom.yaml doesnt push to containers #3410

Closed zebrapatch closed 2 weeks ago

zebrapatch commented 2 weeks ago

Hi, i am running

helm upgrade loki --namespace=prometheus-grafana grafana/loki-stack -f loki-stack-value.yaml

this is my promtail segment of the default loki-stack yaml file.

promtail:
  enabled: true
  config:
    logLevel: info
    serverPort: 3101
    clients:
      - url: http://{{ .Release.Name }}:3100/loki/api/v1/push
    scrape_configs:
      - job_name: 'my_service'  # Name of the job
        scrape_interval: 10s      # Scrape this job every 10 seconds
        static_configs:
          - targets: ['localhost']  # Target to scrape metrics from
      - job_name: audit-logs
        static_configs:
        targets: 
          - localhost
        labels:
          job: audit_logs
          __path__: /var/log/audit/*

running: kubectl exec -it -n prometheus-grafana promtail-2fh4z -- sh this is my etc/promtail/promtail.yaml. Why is this different from the --values (or -f) yaml file that i passed in? (particularly the promtail config portion.)

server:
  log_level: info
  log_format: logfmt
  http_listen_port: 3101

clients:
  - url: http://loki-gateway/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_instance
          - __meta_kubernetes_pod_label_instance
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: instance
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - namespace
        - app
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
        - __meta_kubernetes_pod_container_name
        target_label: __path__

limits_config:

tracing:
  enabled: false

I am trying to use promtail to push custom audit logs from all the nodes. But it is only pushing certain var/log/pods log files. Any help is appreciated.