Log: pipeline stage 'geoip' - values not found in extracted

superstes commented 6 months ago

What's wrong?

The data extracted by the geoip pipeline-stage cannot be referenced in following stages.

I wanted to create different geoip-labels for source- & destination IPs using the template stage (see config section) But it will always return <no data> (or the default-value - in my case).

If one removes the geoip_autonomous_system_number from labeldrop it is added as label, but that does not change the issue.

Did I miss something? This should be possible, or am I wrong?

Related to:

Steps to reproduce

Add a geoip pipeline stage
Try to reference its data in another stage

System information

Docker-Host: Debian 12 Linux 6.5.11-6-pve x86_64, docker-ce: 5:24.0.7-1, container: grafana/agent latest(13 days ago)

Software version

v0.38.1

Configuration

logs:
  configs:
    - name: test
      clients:
        - url: 'http://loki:3100/loki/api/v1/push'
      positions:
        filename: '/tmp/positions.yml'
      scrape_configs:
        - job_name: 'test'
          syslog: {}
        pipeline_stages:
        - match:
            selector: '{facility="kern"} |= "NFTables"'
            stages:
                - regex:
                    expression: '\sSRC=(?P<packet_ip_src>.*?)\s'
                - regex:
                    expression: '\sDST=(?P<packet_ip_dst>.*?)\s'
                - geoip:
                    db: "/etc/geoip/asn.mmdb"
                    source: "packet_ip_src"
                    db_type: "asn"
                - template:
                    source: 'packet_geoip_src_asn'
                    template: '{{or .geoip_autonomous_system_number "-"}}'
                - labeldrop:
                    - 'geoip_autonomous_system_number'

# full: https://github.com/shield-wall-net/controller/blob/latest/files/log/grafana-agent.yml

Logs

No response

tpaschalis commented 6 months ago

Hey there 👋

Yes, you're right. While in Flow mode (using the new config format) the issue was fixed on https://github.com/grafana/agent/pull/4166, the same was not applied on Promtail so that it can be used for static mode (the YAML config).

I'm removing the bug label and applying upstream so we can properly tackle this (the Loki folks would appreciate the contribution as well!) cc @spartan0x117

imtipi commented 5 months ago

Hey there 👋

Yes, you're right. While in Flow mode (using the new config format) the issue was fixed on grafana/agent#4166, the same was not applied on Promtail so that it can be used for static mode (the YAML config).

I'm removing the bug label and applying upstream so we can properly tackle this (the Loki folks would appreciate the contribution as well!) cc @spartan0x117

Hi, https://github.com/grafana/loki/pull/10256 actually fixed this issue,but seems no one reviewed at the moment,it is possible to let team review this pr?

github-actions[bot] commented 4 months ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

rfratto commented 2 months ago

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

grafana / alloy