search

Linuxfabrik / monitoring-plugins

220+ check plugins for Icinga and other Nagios-compatible monitoring applications. Each plugin is a standalone command line tool (written in Python) that provides a specific type of check.
https://linuxfabrik.ch
The Unlicense
220 stars 51 forks source link

docker-stats: ValueError: could not convert string to float: '0B' #776

Closed wwuck closed 2 months ago

wwuck commented 3 months ago

This issue respects the following points:

Which variant of the Monitoring Plugins do you use?

Bug description

I have some VM machines running as gitlab-runner docker hosts, so there are a lot of ephemeral docker containers showing up for a short time period. I have just implemented the docker-stats check on these VM machines, and one of them reported an error.

Traceback (most recent call last):
  File "docker-stats.py", line 264, in 'module'
  File "docker-stats.py", line 206, in main
ValueError: could not convert string to float: '0B'

Steps to reproduce - Plugin call

/usr/lib64/nagios/plugins/docker-stats

Steps to reproduce - Data

No response

Environment

# uname -a
Linux gitlab-runner-01 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

# cat /etc/os-release 
PRETTY_NAME="Devuan GNU/Linux 5 (daedalus)"
NAME="Devuan GNU/Linux"
VERSION_ID="5"
VERSION="5 (daedalus)"
VERSION_CODENAME="daedalus"
ID=devuan
ID_LIKE=debian
HOME_URL="https://www.devuan.org/"
SUPPORT_URL="https://devuan.org/os/community"
BUG_REPORT_URL="https://bugs.devuan.org/"

# cat /etc/debian_version 
12.0

# apt-cache policy docker-ce
docker-ce:
  Installed: 5:27.1.2-1~debian.12~bookworm
  Candidate: 5:27.1.2-1~debian.12~bookworm
  Version table:
 *** 5:27.1.2-1~debian.12~bookworm 500
        500 https://download.docker.com/linux/debian bookworm/stable amd64 Packages
        100 /var/lib/dpkg/status

# docker --version
Docker version 27.1.2, build d01f264

# apt-cache policy linuxfabrik-monitoring-plugins 
linuxfabrik-monitoring-plugins:
  Installed: 2024060401-1
  Candidate: 2024060401-1
  Version table:
 *** 2024060401-1 500
        500 https://repo.linuxfabrik.ch/monitoring-plugins/debian bookworm-release/main amd64 Packages
        100 /var/lib/dpkg/status
     2023112901-1 500
        500 https://repo.linuxfabrik.ch/monitoring-plugins/debian bookworm-release/main amd64 Packages
     2023051201-1 500
        500 https://repo.linuxfabrik.ch/monitoring-plugins/debian bookworm-release/main amd64 Packages

Plugin Version

docker-stats: v2023112901 by Linuxfabrik GmbH, Zurich/Switzerland

Python version

No response

List of Python modules

No response

Additional Information

No response

markuslf commented 3 months ago

What's the output of

wwuck commented 3 months ago

I'm not seeing the error every time I run docker-stats so I can't say how reliable this output will be in relation to hitting the bug.

I ran the docker stats --no-stream within a few seconds of seeing the error on running docker-stats in a terminal window.

# docker info 
Client: Docker Engine - Community
 Version:    27.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 26
  Running: 13
  Paused: 0
  Stopped: 13
 Images: 39
 Server Version: 27.1.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: local
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 6.1.0-23-amd64
 Operating System: Devuan GNU/Linux 5 (daedalus)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.79GiB
 Name: gitlab-runner-01
 ID: d31a7107-754c-48c1-9312-dd2cc6d7abd9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://private-registry-01.internal-domain:5000/
 Live Restore Enabled: false
# docker stats --no-stream
CONTAINER ID   NAME                                                                  CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
628793454d77   runner-ab2l4qzy-project-112-concurrent-0-caeafc5b229959a5-build       52.82%    161.4MiB / 62.79GiB   0.25%     25MB / 420kB      0B / 0B     5
6257cdbe0a15   runner-7ayh6h5f-project-107-concurrent-0-37b2c7aee9359db9-build       95.01%    781.5MiB / 62.79GiB   1.22%     351MB / 1.77MB    0B / 0B     4
0ccb1813299e   runner-7ayh6h5f-project-19-concurrent-0-99f0211c36d59d01-build        59.50%    639.6MiB / 62.79GiB   0.99%     176MB / 16.7MB    0B / 0B     50
c0b966a8386a   runner-ab2l4qzy-project-17-concurrent-0-f36f495faffd695e-build        850.79%   4.699GiB / 62.79GiB   7.48%     197MB / 2.41MB    0B / 0B     757
c41ff5de7977   runner-ab2l4qzy-project-55-concurrent-0-d45818fa754ea730-predefined   97.30%    105.8MiB / 62.79GiB   0.16%     642B / 0B         0B / 0B     21
99d2aef9152e   runner-ab2l4qzy-project-50-concurrent-0-01dbad8d56141494-build        59.35%    54.36MiB / 62.79GiB   0.08%     84.7kB / 8.64kB   0B / 0B     4
f38336200a68   runner-7ayh6h5f-project-49-concurrent-0-e180afe41fc754dc-predefined   79.52%    93.85MiB / 62.79GiB   0.15%     642B / 0B         0B / 0B     21
efb5ad8cc43f   runner-7ayh6h5f-project-54-concurrent-0-9b771e1481f33e66-build        92.19%    74.05MiB / 62.79GiB   0.12%     600B / 0B         0B / 0B     6
               --                                                                    0.00%     0B / 0B               0.00%     0B / 0B           0B / 0B     0
markuslf commented 2 months ago

Output was perfect, thank you. Fixed, please test.

wwuck commented 2 months ago

Looks good here thanks