search

grafana / agent

Vendor-neutral programmable observability pipelines.
https://grafana.com/docs/agent/
Apache License 2.0
1.56k stars 479 forks source link

nil pointer panic when using snmp with v0.40.5 #6927

Closed heathcliff26 closed 3 weeks ago

heathcliff26 commented 1 month ago

What's wrong?

When running grafana-agent docker image v0.40.5 and trying to collect metrics with snmp, the container crashes with a nil pointer exception. Downgrading to v0.40.4 solves the problem.

Steps to reproduce

  1. start grafana-agent container v0.40.5 with snmp enabled

System information

Fedora CoreOS 40.20240504.3.0 with k3s v1.30.0+k3s1

Software version

v0.40.5

Configuration

config-file:

# yaml-language-server: $schema=https://json.schemastore.org/yamllint.json
---
# For a full configuration reference, see: https://grafana.com/docs/agent/latest/configuration/.
server:
  log_level: warn

integrations:
  agent:
    enabled: true
    relabel_configs:
    - replacement: <redacted_hostname>
      target_label: instance
    - replacement: <redacted_hostname>
      target_label: agent_hostname
    - action: replace
      target_label: job
      replacement: "integrations/agent-check"
    metric_relabel_configs:
    - action: keep
      regex: (prometheus_target_.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      - __name__
  prometheus_remote_write:
  - basic_auth:
      password: "${GRAFANA_TOKEN}"
      username: <redacted>
    url: <redacted>

  # SNMP scraping for Synology NAS
  snmp:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels: [job]
        regex: (^.*snmp)\/(.*)
        target_label: job_snmp
        replacement: $1
      - action: replace
        source_labels: [job]
        regex: (^.*snmp)\/(.*)
        target_label: snmp_target
        replacement: $2
      - replacement: <redacted_hostname>
        target_label: instance
      - replacement: <redacted_hostname>
        target_label: 'agent_hostname'
    snmp_targets:
      - name: <redacted_hostname>
        address: <redacted_snmp_client>
        auth: public_v2
        module: if_mib

logs:
  configs:
  - clients:
    - basic_auth:
        password: "${GRAFANA_TOKEN}"
        username: <redacted>
      url: <redacted>
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
    - job_name: integrations/syslog
      syslog:
        listen_address: 0.0.0.0:1514
        listen_protocol: tcp
        idle_timeout: 60s
        label_structured_data: yes
        labels:
          job: "integrations/syslog"
          instance: <redacted_hostname>
      relabel_configs:
        - source_labels: ['__syslog_message_hostname']
          target_label: 'host'
        - replacement: <redacted_hostname>
          target_label: 'agent_hostname'

deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: syslog-agent
  namespace: grafana-syslog
  labels:
    app: syslog-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: syslog-agent
  template:
    metadata:
      labels:
        app: syslog-agent
    spec:
      containers:
        - name: grafana-agent
          envFrom:
            - secretRef:
                name: syslog-agent-diskstation
          image: docker.io/grafana/agent:v0.40.4
          args: ["--disable-reporting", "--config.file=/etc/agent/agent.yaml","--config.expand-env" ,"--metrics.wal-directory=/etc/agent/data"]
          ports:
            - name: syslog
              containerPort: 1514
              protocol: TCP
          volumeMounts:
            - name: grafana-agent-config
              mountPath: /etc/agent/agent.yaml
              subPath: agent.yaml
      volumes:
        - name: grafana-agent-config
          configMap:
            name: syslog-agent
            items:
              - key: grafana-agent.yaml
                path: agent.yaml

Logs

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x585cc25]
goroutine 665 [running]:
github.com/prometheus/snmp_exporter/collector.Collector.collect({{0x8ca1848, 0xc0031b0b40}, {0xc0049e8403, 0x14}, 0xc0023b0120, {0xc0049e83e4, 0x9}, {0xc0049526f0, 0x1, 0x1}, ...}, ...)
 /go/pkg/mod/github.com/prometheus/snmp_exporter@v0.26.0/collector/collector.go:363 +0x305
github.com/prometheus/snmp_exporter/collector.Collector.Collect.func1(0xc001ba1a40?)
 /go/pkg/mod/github.com/prometheus/snmp_exporter@v0.26.0/collector/collector.go:458 +0x8d1
created by github.com/prometheus/snmp_exporter/collector.Collector.Collect in goroutine 664
 /go/pkg/mod/github.com/prometheus/snmp_exporter@v0.26.0/collector/collector.go:432 +0x13d
db-wally007 commented 4 weeks ago

same here - simple upgrade of the podman container fails (I'm using flow configuration)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x581d065]

goroutine 723 [running]:
github.com/prometheus/snmp_exporter/collector.Collector.collect({{0x8c55e10, 0xc004c04a80}, {0xc005df40de, 0xb}, 0xc001cb4870, {0x79a6bb0, 0x9}, {0xc001ba42f8, 0x1, 0x1}, ...}, ...)
        /go/pkg/mod/github.com/prometheus/snmp_exporter@v0.26.0/collector/collector.go:363 +0x305
github.com/prometheus/snmp_exporter/collector.Collector.Collect.func1(0xc003376440?)
        /go/pkg/mod/github.com/prometheus/snmp_exporter@v0.26.0/collector/collector.go:458 +0x8d1
created by github.com/prometheus/snmp_exporter/collector.Collector.Collect in goroutine 722
        /go/pkg/mod/github.com/prometheus/snmp_exporter@v0.26.0/collector/collector.go:432 +0x13d

Rollback to 0.40.4 = works fine.

ptodev commented 3 weeks ago

Hi, apologies for this issue. It was fixed in v0.41.0 via #6921

ptodev commented 5 days ago

Hi @db-wally007! Do you mind sharing your Agent config file please (with secrets redacted)? I just tried v0.41.1 with this config, and it worked without issues:

server:
  log_level: debug
integrations:
  snmp:
    enabled: true
    scrape_interval: 1m
    scrape_timeout: 30s
    snmp_targets:
      - name: testing
        address: demo.pysnmp.com
        module: if_mib
        auth: public_v2
db-wally007 commented 5 days ago

Hi @db-wally007! Do you mind sharing your Agent config file please (with secrets redacted)? I just tried v0.41.1 with this config, and it worked without issues:

server:
  log_level: debug
integrations:
  snmp:
    enabled: true
    scrape_interval: 1m
    scrape_timeout: 30s
    snmp_targets:
      - name: testing
        address: demo.pysnmp.com
        module: if_mib
        auth: public_v2

I found that the issue was with our CD/CI - or rather Grafana repository not hosting latest grafana-agent-flow rpm.

During container process we install latest grafana-agent-flow rpm directly from Grafana's RPM repo - but for some reason "flow" x86-64 rpm version is missing.

Any reason why "flow" version wasnt pushed ? Repo: https://rpm.grafana.com

root@podman2:~>yum --disablerepo="*" --enablerepo="grafana" list available
Updating Subscription Management repositories.
Last metadata expiration check: 0:22:02 ago on Fri 21 Jun 2024 08:31:02 AM UTC.
Available Packages
alloy.aarch64                                                                                                      1.1.1-1                                                                                             grafana
alloy.ppc64le                                                                                                      1.1.1-1                                                                                             grafana
alloy.s390x                                                                                                        1.1.1-1                                                                                             grafana
alloy.x86_64                                                                                                       1.1.1-1                                                                                             grafana
grafana.armhfp                                                                                                     10.0.3-1                                                                                            grafana
grafana.armhf                                                                                                      10.1.2-1                                                                                            grafana
grafana.aarch64                                                                                                    11.0.0-1                                                                                            grafana
grafana.x86_64                                                                                                     11.0.0-1                                                                                            grafana
grafana-agent.armhfp                                                                                               0.32.1-1                                                                                            grafana
grafana-agent.aarch64                                                                                              0.41.1-1                                                                                            grafana
grafana-agent.ppc64le                                                                                              0.41.1-1                                                                                            grafana
grafana-agent.s390x                                                                                                0.41.1-1                                                                                            grafana
grafana-agent.x86_64                                                                                               0.41.1-1                                                                                            grafana
grafana-agent-flow.aarch64                                                                                         0.41.1-1                                                                                            grafana
grafana-agent-flow.ppc64le                                                                                         0.41.1-1                                                                                            grafana
grafana-agent-flow.s390x                                                                                           0.41.1-1                                                                                            grafana
grafana-enterprise.armhfp                                                                                          10.0.3-1                                                                                            grafana
grafana-enterprise.armhf                                                                                           10.1.2-1                                                                                            grafana
grafana-enterprise.aarch64                                                                                         11.0.0-1                                                                                            grafana
grafana-enterprise.x86_64                                                                                          11.0.0-1                                                                                            grafana
grafana-enterprise-nightly.armhf                                                                                   10.3.0_62843-1                                                                                      grafana
grafana-enterprise-nightly.aarch64                                                                                 11.2.0_72310-1                                                                                      grafana
grafana-enterprise-nightly.x86_64                                                                                  11.2.0_72310-1                                                                                      grafana
grafana-nightly.armhf                                                                                              10.3.0_143857-1                                                                                     grafana
grafana-nightly.aarch64                                                                                            11.2.0_184028-1                                                                                     grafana
grafana-nightly.x86_64                                                                                             11.2.0_184028-1                                                                                     grafana
logcli.aarch64                                                                                                     3.0.0-1                                                                                             grafana
logcli.arm                                                                                                         3.0.0-1                                                                                             grafana
logcli.x86_64                                                                                                      3.0.0-1                                                                                             grafana
loki.aarch64                                                                                                       3.0.0-1                                                                                             grafana
loki.arm                                                                                                           3.0.0-1                                                                                             grafana
loki.x86_64                                                                                                        3.0.0-1                                                                                             grafana
loki-canary.aarch64                                                                                                3.0.0-1                                                                                             grafana
loki-canary.arm                                                                                                    3.0.0-1                                                                                             grafana
loki-canary.x86_64                                                                                                 3.0.0-1                                                                                             grafana
metaconvert.aarch64                                                                                                2.12.0-1                                                                                            grafana
metaconvert.x86_64                                                                                                 2.12.0-1                                                                                            grafana
mimir.aarch64                                                                                                      2.12.0-1                                                                                            grafana
mimir.x86_64                                                                                                       2.12.0-1                                                                                            grafana
mimir-continuous-test.aarch64                                                                                      2.12.0-1                                                                                            grafana
mimir-continuous-test.x86_64                                                                                       2.12.0-1                                                                                            grafana
mimirtool.aarch64                                                                                                  2.12.0-1                                                                                            grafana
mimirtool.x86_64                                                                                                   2.12.0-1                                                                                            grafana
promtail.aarch64                                                                                                   3.0.0-1                                                                                             grafana
promtail.arm                                                                                                       3.0.0-1                                                                                             grafana
promtail.x86_64                                                                                                    3.0.0-1                                                                                             grafana
pyroscope.aarch64                                                                                                  1.6.0-1                                                                                             grafana
pyroscope.armv6hl                                                                                                  1.6.0-1                                                                                             grafana
pyroscope.armv7hl                                                                                                  1.6.0-1                                                                                             grafana
pyroscope.x86_64                                                                                                   1.6.0-1                                                                                             grafana
query-tee.aarch64                                                                                                  2.12.0-1                                                                                            grafana
query-tee.x86_64                                                                                                   2.12.0-1                                                                                            grafana
synthetic-monitoring-agent.arm                                                                                     0.9.4_0_g18710e2-1                                                                                  grafana
synthetic-monitoring-agent.armv6hl                                                                                 0.17.3-1                                                                                            grafana
synthetic-monitoring-agent.aarch64                                                                                 0.24.3-1                                                                                            grafana
synthetic-monitoring-agent.x86_64                                                                                  0.24.3-1                                                                                            grafana
tempo.aarch64                                                                                                      2.5.0-1                                                                                             grafana
tempo.armv6hl                                                                                                      2.5.0-1                                                                                             grafana
tempo.x86_64                                                                                                       2.5.0-1
ptodev commented 5 days ago

Thank you! Would you be able to tell me what the version of grafana-agent.x86_64 on the rpm repo is please? I'm on a Mac and I'm not sure how I could check it on my own.

I had a look at the CI for the latest release, and I get the impression that everything was successfully published.

db-wally007 commented 5 days ago

Thank you! Would you be able to tell me what the version of grafana-agent.x86_64 on the rpm repo is please? I'm on a Mac and I'm not sure how I could check it on my own.

I had a look at the CI for the latest release, and I get the impression that everything was successfully published.

grafana-agent is there (see my post above) grafana-agent-flow is missing for x86-64 (aarch64,ppc64le and s390x is there)

this is the list 

root@podman2:~>yum --disablerepo="*" --enablerepo="grafana" list available
Updating Subscription Management repositories.
Last metadata expiration check: 0:22:02 ago on Fri 21 Jun 2024 08:31:02 AM UTC.
Available Packages
alloy.aarch64                                                                                                      1.1.1-1                                                                                             grafana
alloy.ppc64le                                                                                                      1.1.1-1                                                                                             grafana
alloy.s390x                                                                                                        1.1.1-1                                                                                             grafana
alloy.x86_64                                                                                                       1.1.1-1                                                                                             grafana
grafana.armhfp                                                                                                     10.0.3-1                                                                                            grafana
grafana.armhf                                                                                                      10.1.2-1                                                                                            grafana
grafana.aarch64                                                                                                    11.0.0-1                                                                                            grafana
grafana.x86_64                                                                                                     11.0.0-1                                                                                            grafana
grafana-agent.armhfp                                                                                               0.32.1-1                                                                                            grafana
grafana-agent.aarch64                                                                                              0.41.1-1                                                                                            grafana
grafana-agent.ppc64le                                                                                              0.41.1-1                                                                                            grafana
grafana-agent.s390x                                                                                                0.41.1-1                                                                                            grafana
grafana-agent.x86_64                                                                                               0.41.1-1                                                                                            grafana
grafana-agent-flow.aarch64                                                                                         0.41.1-1                                                                                            grafana
grafana-agent-flow.ppc64le                                                                                         0.41.1-1                                                                                            grafana
grafana-agent-flow.s390x                                                                                           0.41.1-1                                                                                            grafana
grafana-enterprise.armhfp                                                                                          10.0.3-1                                                                                            grafana
grafana-enterprise.armhf                                                                                           10.1.2-1                                                                                            grafana
grafana-enterprise.aarch64                                                                                         11.0.0-1                                                                                            grafana
grafana-enterprise.x86_64                                                                                          11.0.0-1                                                                                            grafana
grafana-enterprise-nightly.armhf                                                                                   10.3.0_62843-1                                                                                      grafana
grafana-enterprise-nightly.aarch64                                                                                 11.2.0_72310-1                                                                                      grafana
grafana-enterprise-nightly.x86_64                                                                                  11.2.0_72310-1                                                                                      grafana
grafana-nightly.armhf                                                                                              10.3.0_143857-1                                                                                     grafana
grafana-nightly.aarch64                                                                                            11.2.0_184028-1                                                                                     grafana
grafana-nightly.x86_64                                                                                             11.2.0_184028-1                                                                                     grafana
logcli.aarch64                                                                                                     3.0.0-1                                                                                             grafana
logcli.arm                                                                                                         3.0.0-1                                                                                             grafana
logcli.x86_64                                                                                                      3.0.0-1                                                                                             grafana
loki.aarch64                                                                                                       3.0.0-1                                                                                             grafana
loki.arm                                                                                                           3.0.0-1                                                                                             grafana
loki.x86_64                                                                                                        3.0.0-1                                                                                             grafana
loki-canary.aarch64                                                                                                3.0.0-1                                                                                             grafana
loki-canary.arm                                                                                                    3.0.0-1                                                                                             grafana
loki-canary.x86_64                                                                                                 3.0.0-1                                                                                             grafana
metaconvert.aarch64                                                                                                2.12.0-1                                                                                            grafana
metaconvert.x86_64                                                                                                 2.12.0-1                                                                                            grafana
mimir.aarch64                                                                                                      2.12.0-1                                                                                            grafana
mimir.x86_64                                                                                                       2.12.0-1                                                                                            grafana
mimir-continuous-test.aarch64                                                                                      2.12.0-1                                                                                            grafana
mimir-continuous-test.x86_64                                                                                       2.12.0-1                                                                                            grafana
mimirtool.aarch64                                                                                                  2.12.0-1                                                                                            grafana
mimirtool.x86_64                                                                                                   2.12.0-1                                                                                            grafana
promtail.aarch64                                                                                                   3.0.0-1                                                                                             grafana
promtail.arm                                                                                                       3.0.0-1                                                                                             grafana
promtail.x86_64                                                                                                    3.0.0-1                                                                                             grafana
pyroscope.aarch64                                                                                                  1.6.0-1                                                                                             grafana
pyroscope.armv6hl                                                                                                  1.6.0-1                                                                                             grafana
pyroscope.armv7hl                                                                                                  1.6.0-1                                                                                             grafana
pyroscope.x86_64                                                                                                   1.6.0-1                                                                                             grafana
query-tee.aarch64                                                                                                  2.12.0-1                                                                                            grafana
query-tee.x86_64                                                                                                   2.12.0-1                                                                                            grafana
synthetic-monitoring-agent.arm                                                                                     0.9.4_0_g18710e2-1                                                                                  grafana
synthetic-monitoring-agent.armv6hl                                                                                 0.17.3-1                                                                                            grafana
synthetic-monitoring-agent.aarch64                                                                                 0.24.3-1                                                                                            grafana
synthetic-monitoring-agent.x86_64                                                                                  0.24.3-1                                                                                            grafana
tempo.aarch64                                                                                                      2.5.0-1                                                                                             grafana
tempo.armv6hl                                                                                                      2.5.0-1                                                                                             grafana
tempo.x86_64                                                                                                       2.5.0-1