search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.58k stars 3.41k forks source link

Promtail hi cpu usage in 2.6.1 #7111

Open mortaelth opened 2 years ago

mortaelth commented 2 years ago

Describe the bug After upgrade to Promtail 2.6.1 from 2.4.1, Promtail takes all the nodes CPU (all nodes running on 80-100%). When version is downgraded, CPU gets to normal.

On our 8-core machine, promtail eats more then 6 cores. Version 2.4.1 using less then 0.1 core

It is connected with the pipeline config we have in Promtail for parsing some information from logs. If I remove the "kubernetes-custom-logs-app" and "kubernetes-custom-logs" jobs (see configuration below), the CPU in 2.6.1 is similar to levels in 2.4.1. We know the regular expressions are cpu-heavy, we are ok with "some" increase as we need to parse those information. To be honest the CPU levels in 2.4.1 are VERY low in respect to the configuration.

To Reproduce Steps to reproduce the behavior:

  1. Upgraded Promtail from 2.4.1 to 2.6.1

Expected behavior CPU in the levels as on v2.4.1

Environment:

Screenshots, Promtail config, or terminal output


clients:
  - external_labels:
        cluster: test3
    backoff_config:
        min_period: 500ms
        max_period: 5m
        max_retries: 11
    batchsize: 304800
    batchwait: 2s
    url: http://distributor.loki.svc.cluster.local:3100/loki/api/v1/push
    tenant_id: 1

scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      labels:
        job: journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__hostname']
        target_label: nodename
  - job_name: kubernetes-pods-name
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - labeldrop:
        - filename
        - subCluster
        - controller_revision_hash
        - app_kubernetes_io_managed_by
        - statefulset_kubernetes_io_pod_name
        - stream
        - pod_template_hash
        - pod_template_generation
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_label_name
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - __meta_kubernetes_namespace
          - __service__
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__

  - job_name: kubernetes-custom-logs-app
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - drop:
          expression: "^JLOG"
      - labeldrop:
          - filename
          - subCluster
          - controller_revision_hash
          - app_kubernetes_io_managed_by
          - statefulset_kubernetes_io_pod_name
          - stream
          - pod_template_hash
          - pod_template_generation
    relabel_configs:
      - source_labels: 
          - __meta_kubernetes_pod_label_app
        action: keep
        regex: hornet-processor|wasp-scripter
      - action: drop
        regex: .+
        source_labels:
          - __meta_kubernetes_pod_label_name
      - source_labels:
          - __meta_kubernetes_pod_label_app
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - __meta_kubernetes_namespace
          - __service__
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
  - job_name: kubernetes-custom-logs
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - drop:
          expression: "^(.{0,4}$|[^J].+|J[^L].+|JL[^O].+|JLO[^G].+)"
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GOD_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GOD_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+)(?:,serverId:[^,]+,)(?P<output>(.+))(?::JLOG_GOD_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '1'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GB_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GB_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+)(?:,serverId:[^,]+,)(?P<output>(.+))(?::JLOG_GB_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '2'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GOD_H_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GOD_H_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>(.+))(?::JLOG_GOD_H_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '100'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GB_H_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GB_H_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>(.+))(?::JLOG_GB_H_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '200'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GOD_SUCC_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GOD_SUCC_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>(.+))(?::JLOG_GOD_SUCC_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '101'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GB_SUCC_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GB_SUCC_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>(.+))(?::JLOG_GB_SUCC_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '201'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GOD_FAIL_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GOD_FAIL_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>(.+))(?::JLOG_GOD_FAIL_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '102'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GB_FAIL_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GB_FAIL_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>(.+))(?::JLOG_GB_FAIL_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '202'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GOD_DBG_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GOD_DBG_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>.*)(?::JLOG_GOD_DBG_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '103'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GOD_DBG_H_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GOD_DBG_H_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>.*)(?::JLOG_GOD_DBG_H_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '104'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GB_DBG_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GB_DBG_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>.*)(?::JLOG_GB_DBG_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '203'
          - labels:
              job_log:
              companyId:
      - match: 
          selector: '{app=~"hornet-processor|wasp-scripter"} |~ "^(JLOG_GB_DBG_H_LOKI_START)"'
          stages:
          - regex:
              expression: "^(?P<job_log>JLOG_GB_DBG_H_LOKI_START:)(?:companyId:)(?P<companyId>[0-9]+),(?P<output>.*)(?::JLOG_GB_DBG_H_LOKI_END)"
          - output:
              source: output
          - template:
              source: job_log 
              template: '204'
          - labels:
              job_log:
              companyId:
      - labeldrop:
          - filename
          - subCluster
          - controller_revision_hash
          - app_kubernetes_io_managed_by
          - statefulset_kubernetes_io_pod_name
          - stream
          - pod_template_hash
          - pod_template_generation
          - deploymentName
          - configHash
    relabel_configs:
      - source_labels: 
          - __meta_kubernetes_pod_label_app
        action: keep
        regex: hornet-processor|wasp-scripter
      - action: drop
        regex: .+
        source_labels:
          - __meta_kubernetes_pod_label_name
      - source_labels:
          - __meta_kubernetes_pod_label_app
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__

  - job_name: kubernetes-pods-app
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - match: 
          selector: '{app="eventrouter"}'
          stages:
          - json:
              expressions:
                event_namespace: event.metadata.namespace
          - labels:
              event_namespace:
      - labeldrop:
        - filename
        - subCluster
        - controller_revision_hash
        - app_kubernetes_io_managed_by
        - statefulset_kubernetes_io_pod_name
        - stream
        - pod_template_hash
        - pod_template_generation
    relabel_configs:
      - source_labels: 
          - __meta_kubernetes_pod_label_app
        action: drop
        regex: wasp-scripter|hornet-processor
      - action: drop
        regex: .+
        source_labels:
          - __meta_kubernetes_pod_label_name
      - source_labels:
          - __meta_kubernetes_pod_label_app
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - __meta_kubernetes_namespace
          - __service__
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
  - job_name: kubernetes-pods-direct-controllers
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - labeldrop:
        - filename
        - subCluster
        - controller_revision_hash
        - app_kubernetes_io_managed_by
        - statefulset_kubernetes_io_pod_name
        - stream
        - pod_template_hash
        - pod_template_generation
    relabel_configs:
      - action: drop
        regex: .+
        separator: ""
        source_labels:
          - __meta_kubernetes_pod_label_name
          - __meta_kubernetes_pod_label_app
      - action: drop
        regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
        source_labels:
          - __meta_kubernetes_pod_controller_name
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - __meta_kubernetes_namespace
          - __service__
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
  - job_name: kubernetes-pods-indirect-controller
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - labeldrop:
        - filename
        - subCluster
        - controller_revision_hash
        - app_kubernetes_io_managed_by
        - statefulset_kubernetes_io_pod_name
        - stream
        - pod_template_hash
        - pod_template_generation
    relabel_configs:
      - action: drop
        regex: .+
        separator: ""
        source_labels:
          - __meta_kubernetes_pod_label_name
          - __meta_kubernetes_pod_label_app
      - action: keep
        regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
        source_labels:
          - __meta_kubernetes_pod_controller_name
      - action: replace
        regex: ([0-9a-z-.]+)-[0-9a-f]{8,10}
        source_labels:
          - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - __meta_kubernetes_namespace
          - __service__
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
  - job_name: kubernetes-pods-static
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - labeldrop:
        - filename
    relabel_configs:
      - action: drop
        regex: ""
        source_labels:
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_label_component
        target_label: __service__
      - source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ""
        source_labels:
          - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - __meta_kubernetes_namespace
          - __service__
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
          - __meta_kubernetes_pod_container_name
        target_label: __path__
craftyc0der commented 2 years ago

This is a known issue. There was a patch made, but it has not been released. I tried using it, and stopped capturing logs for my trouble, so I think the patch is not ready yet.

mortaelth commented 2 years ago

This is a known issue. There was a patch made, but it has not been released. I tried using it, and stopped capturing logs for my trouble, so I think the patch is not ready yet.

Thank you, could you please reference the issue you are mentioning?

mortaelth commented 2 years ago

Hello all, this is still a blocker for us to update to the latest Promtail version. I will be happy to try something or dig in it more. At least when trying the config without the regex matches with positive look-ahead, the cpu looks much better. Any idea what was changed between 2.4.1 and 2.6.1 what could affect this?

mortaelth commented 1 year ago

issue still present