prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.23k stars 2.36k forks source link

FreeBSD: couldn't get uname: cannot allocate memory #2809

Open pando85 opened 1 year ago

pando85 commented 1 year ago

Host operating system: output of uname -a

FreeBSD XXXX 14.0-CURRENT FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/amd64.amd64/sys/pfSense amd64

node_exporter version: output of node_exporter --version

node_exporter, version 1.5.0 (branch: release-1.5, revision: 5)
  build user:       nobody
  build date:       
  go version:       go1.20.3
  platform:         freebsd/amd64

node_exporter command line flags

/usr/local/bin/node_exporter --web.listen-address=192.168.192.1:9100 --collector.textfile.directory=/var/tmp/node_exporter --collector.boottime --collector.cpu --collector.exec --collector.filesystem --collector.loadavg --collector.meminfo --collector.netdev --collector.textfile --collector.time --log.level=warn

node_exporter log output

s=2023-09-22T20:05:32.809Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=0.000157114 err="cannot allocate memory" 

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Run node_exporter.

What did you expect to see?

node_uname_info metric.

What did you see instead?

node_scrape_collector_success{collector="uname"} 0
MarshmallowSoup commented 1 year ago

Same here

discordianfish commented 1 year ago

I suppose this comes from here: https://github.com/prometheus/node_exporter/blob/7dd2f0dc4f632f7778be134ace0e7c8ecdfe279a/collector/uname_bsd.go#L28

Can create a minimal program using unix.Uname() to reproduce this? Might be a golang issue. Also, have you tried the current master version?

joelsdc commented 10 months ago

Happening also on latest pfSense OSS v2.7.2 (based on FreeBSD 14)

discordianfish commented 9 months ago

Please anyone try to reproduce this by running the following:

package main

import (
    "log"
    "golang.org/x/sys/unix"
)

func main() {
    var utsname unix.Utsname
    if err := unix.Uname(&utsname); err != nil {
        log.Fatal(err)
    }
    log.Printf("%s", utsname)
}
seeplusplus commented 9 months ago

Please anyone try to reproduce this by running the following:

package main

import (
  "log"
  "golang.org/x/sys/unix"
)

func main() {
  var utsname unix.Utsname
  if err := unix.Uname(&utsname); err != nil {
      log.Fatal(err)
  }
  log.Printf("%s", utsname)
}

Yes, this reproduces the exact same input described above:

2024/02/07 17:11:22 cannot allocate memory
seeplusplus commented 9 months ago

I have tested, by building the above with the entirety of https://cs.opensource.google/go/x/sys/+/master:unix/syscall_freebsd.go;l=169-213;bpv=0?q=Uname&ss=go%2Fx%2Fsys commented out and progressively uncommenting line by line and running each executable as I go along and have determined that this line chunk:

         mib = []_C_int{CTL_KERN, KERN_VERSION}
          n = unsafe.Sizeof(uname.Version)
          if err := sysctl(mib, &uname.Version[0], &n, nil, 0); err != nil {
                 return err
}   

to be the cause. Your minimal repro code snippet works if I leave everything but this section commented out, and continues to fail to alloc if I leave this chunk in. I have never written Go, so I'm not sure if I'll be able to fix this but wanted to drop this here in case anyone else lands here.

Edit

https://man.freebsd.org/cgi/man.cgi?sysctl(3)

https://cs.opensource.google/go/x/sys/+/master:unix/ztypes_freebsd_amd64.go;l=639-645;bpv=0?q=Utsname&ss=go%2Fx%2Fsys

This is because the Version field of Utsname is too small. It's 256 bytes above, but in pfsense this string is larger.

discordianfish commented 9 months ago

@seeplusplus Great job debugging and thanks for opening the golang issue!

seeplusplus commented 9 months ago

@seeplusplus Great job debugging and thanks for opening the golang issue!

The fix for the issue in sys/unix was merged to master this morning. It seems like there are some other issues in this repo preventing successful builds on FreeBSD though. I started taking a look a those over the weekend.