search

prometheus / procfs

procfs provides functions to retrieve system, kernel and process metrics from the pseudo-filesystem proc.
Apache License 2.0
756 stars 311 forks source link

Couldn't parse \"/proc/stat\": bufio.Scanner: token too long #546

Closed drdeimos closed 1 year ago

drdeimos commented 1 year ago

Host operating system:

Linux k42jv 6.4.2 prometheus/node_exporter#1 SMP PREEMPT_DYNAMIC Thu Jul 6 15:29:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version:

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962)
  build user:       root@f9c3ed0cfbd3
  build date:       20230527-12:03:54
  go version:       go1.20.4
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9101 --collector.netdev.address-info --collector.ntp --collector.ntp.server-is-local --collector.filesystem.mount-points-exclude (^/(dev|proc|sys|run|var/lib/kubelet)($|/))|(^/var/lib/docker/) --collector.filesystem.fs-types-exclude ^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|fuse\.lxcfs|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs|squashfs)$

node_exporter log output

log ``` Jul 07 21:58:49 k42jv systemd[1]: Started Prometheus exporter for hardware and OS metrics. Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.626Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.6.0, branch=HEAD, revision=ff7f9d69b645cb691dd3e84dc3afc88f5c006962)" Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.635Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.20.4, platform=linux/amd64, user=root@f9c3ed0cfbd3, date=20230527-12:03:54, tags=netgo osusergo static_build)" Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.636Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.638Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=(^/(dev|proc|sys|run|var/lib/kubelet)($|/))|(^/var/lib/docker/) Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|fuse\.lxcfs|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs|squashfs)$ Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=ntp.go:83 level=warn collector=ntp msg="This collector is deprecated and will be removed in the next major version release." Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:110 level=info msg="Enabled collectors" Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=arp Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=bcache Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=bonding Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=btrfs Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=conntrack Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=cpu Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=cpufreq Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=diskstats Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=dmi Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=edac Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=entropy Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=fibrechannel Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=filefd Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=filesystem Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=hwmon Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=infiniband Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=ipvs Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=loadavg Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=mdadm Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=meminfo Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=netclass Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=netdev Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=netstat Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=nfs Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=nfsd Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=ntp Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=nvme Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=os Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=powersupplyclass Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=pressure Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=rapl Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=schedstat Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=selinux Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=sockstat Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=softnet Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=stat Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=tapestats Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=textfile Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=thermal_zone Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=time Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=timex Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=udp_queues Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=uname Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=vmstat Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=xfs Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=zfs Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.660Z caller=tls_config.go:274 level=info msg="Listening on" address=127.0.0.1:9101 Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.660Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9101 Jul 07 21:59:09 k42jv node_exporter[484]: ts=2023-07-07T18:59:09.457Z caller=collector.go:169 level=error msg="collector failed" name=cpu duration_seconds=0.006603211 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long" Jul 07 21:59:09 k42jv node_exporter[484]: ts=2023-07-07T18:59:09.459Z caller=collector.go:169 level=error msg="collector failed" name=stat duration_seconds=0.007868332 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long" Jul 07 21:59:24 k42jv node_exporter[484]: ts=2023-07-07T18:59:24.443Z caller=collector.go:169 level=error msg="collector failed" name=cpu duration_seconds=0.000539895 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long" Jul 07 21:59:24 k42jv node_exporter[484]: ts=2023-07-07T18:59:24.446Z caller=collector.go:169 level=error msg="collector failed" name=stat duration_seconds=0.001681038 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long" Jul 07 21:59:39 k42jv node_exporter[484]: ts=2023-07-07T18:59:39.440Z caller=collector.go:169 level=error msg="collector failed" name=cpu duration_seconds=0.001306421 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long" ```

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Run node_exporter on my laptop with latest kernels

What did you expect to see?

I don't see some metrics like node_boot_time_seconds or node_cpu_seconds_total

What did you see instead?

Nothing. Only log above

drdeimos commented 1 year ago

That's content of /proc/stat from this machine proc.stat.txt

SuperQ commented 1 year ago

This is a parsing issue in procfs. Moving to the correct repo.

SuperQ commented 1 year ago

Wow, there are 65962 columns in that intr line. The default Scanner can only handle 65535 columns. Easy enough to fix.

Joseph-co commented 3 months ago

Wow, there are 65962 columns in that intr line. The default Scanner can only handle 65535 columns. Easy enough to fix.

Could you tell me how to increase those parses, "Increase the /proc/stat scanner tokens size from the default of 65k to 1024k." and "Increase the scanner buffer from the default 4kB to 8kB bytes". How to implement stuff.