blkio error

[tolotos-rgu]

My syslog is full of these lines after activating this plugin:

Unexpected number of blkio stats for container 16c4e37/sharp_lamarr!

Is this a configuration error on my side?

{u'blkio_stats': {u'io_merged_recursive': [], u'io_queue_recursive': [], u'io_service_bytes_recursive': [{u'major': 8, u'minor': 16, u'op': u'Read', u'value': 230921568256}, {u'major': 8, u'minor': 16, u'op': u'Write', u'value': 0}, {u'major': 8, u'minor': 16, u'op': u'Sync', u'value': 230921568256}, {u'major': 8, u'minor': 16, u'op': u'Async', u'value': 0}, {u'major': 8, u'minor': 16, u'op': u'Discard', u'value': 0}, {u'major': 8, u'minor': 16, u'op': u'Total', u'value': 230921568256}, {u'major': 8, u'minor': 0, u'op': u'Read', u'value': 2069770240}, {u'major': 8, u'minor': 0, u'op': u'Write', u'value': 13234159616}, {u'major': 8, u'minor': 0, u'op': u'Sync', u'value': 12411502592}, {u'major': 8, u'minor': 0, u'op': u'Async', u'value': 2892427264}, {u'major': 8, u'minor': 0, u'op': u'Discard', u'value': 0}, {u'major': 8, u'minor': 0, u'op': u'Total', u'value': 15303929856}], u'io_service_time_recursive': [], u'io_serviced_recursive': [{u'major': 8, u'minor': 16, u'op': u'Read', u'value': 579063}, {u'major': 8, u'minor': 16, u'op': u'Write', u'value': 0}, {u'major': 8, u'minor': 16, u'op': u'Sync', u'value': 579063}, {u'major': 8, u'minor': 16, u'op': u'Async', u'value': 0}, {u'major': 8, u'minor': 16, u'op': u'Discard', u'value': 0}, {u'major': 8, u'minor': 16, u'op': u'Total', u'value': 579063}, {u'major': 8, u'minor': 0, u'op': u'Read', u'value': 274004}, {u'major': 8, u'minor': 0, u'op': u'Write', u'value': 1205249}, {u'major': 8, u'minor': 0, u'op': u'Sync', u'value': 1378476}, {u'major': 8, u'minor': 0, u'op': u'Async', u'value': 100777}, {u'major': 8, u'minor': 0, u'op': u'Discard', u'value': 0}, {u'major': 8, u'minor': 0, u'op': u'Total', u'value': 1479253}], u'io_time_recursive': [], u'io_wait_time_recursive': [], u'sectors_recursive': []}, u'cpu_stats': {u'cpu_usage': {u'percpu_usage': [6821925991403, 7244732900807, 7122472098723, 7039060391479, 7035956744408, 6905397653626], u'total_usage': 42169545780446, u'usage_in_kernelmode': 4623820000000, u'usage_in_usermode': 36548740000000}, u'online_cpus': 6, u'system_cpu_usage': 1501104980000000, u'throttling_data': {u'periods': 0, u'throttled_periods': 0, u'throttled_time': 0}}, u'id': u'16c4e37edf29c601d864058b7c446e3304143f9bac335f4fa881869b25dac04d', u'memory_stats': {u'limit': 16821358592, u'max_usage': 15919288320, u'stats': {u'active_anon': 154034176, u'active_file': 241438720, u'cache': 370274304, u'dirty': 270336, u'hierarchical_memory_limit': 9223372036854771712, u'hierarchical_memsw_limit': 0, u'inactive_anon': 157753344, u'inactive_file': 114147328, u'mapped_file': 45551616, u'pgfault': 752315685, u'pgmajfault': 5148, u'pgpgin': 749260281, u'pgpgout': 749319112, u'rss': 297885696, u'rss_huge': 159383552, u'total_active_anon': 154034176, u'total_active_file': 241438720, u'total_cache': 370274304, u'total_dirty': 270336, u'total_inactive_anon': 157753344, u'total_inactive_file': 114147328, u'total_mapped_file': 45551616, u'total_pgfault': 752315685, u'total_pgmajfault': 5148, u'total_pgpgin': 749260281, u'total_pgpgout': 749319112, u'total_rss': 297885696, u'total_rss_huge': 159383552, u'total_unevictable': 4096, u'total_writeback': 270336, u'unevictable': 4096, u'writeback': 270336}, u'usage': 745459712}, u'name': u'/sharp_lamarr', u'networks': {u'eth0': {u'rx_bytes': 772860979, u'rx_dropped': 0, u'rx_errors': 0, u'rx_packets': 933685, u'tx_bytes': 5278429071, u'tx_dropped': 0, u'tx_errors': 0, u'tx_packets': 675108}}, u'num_procs': 0, u'pids_stats': {u'current': 33}, u'precpu_stats': {u'cpu_usage': {u'total_usage': 0, u'usage_in_kernelmode': 0, u'usage_in_usermode': 0}, u'throttling_data': {u'periods': 0, u'throttled_periods': 0, u'throttled_time': 0}}, u'preread': u'0001-01-01T00:00:00Z', u'read': u'2019-01-19T14:31:32.159411822Z', u'storage_stats': {}}

[unix196]

I confirm, we also have many messages, like

oct 14 08:14:53 asrv241 collectd[27662]: docker : Unexpected number of blkio stats for container 86bb404/brubeck!
oct 14 08:14:53 asrv241 collectd[27662]: docker : Unexpected number of blkio stats for container 068037f/netdata!

Our config:

cat /etc/collectd/plugins/docker.conf 
<LoadPlugin python>
    Globals true
</LoadPlugin>

<Plugin python>
    ModulePath "/usr/share/collectd/modules"
    LogTraces true
    Interactive false

    Import "dockerplugin"
    <Module "dockerplugin">
        BaseURL "unix://var/run/docker.sock"
        Timeout 3
    </Module>
</Plugin>

Debian GNU/Linux 9.6 (stretch)
ii  docker-ce                        5:18.09.0~3-0~debian-stretch             amd64        Docker: the open-source application container engine
ii  docker-ce-cli                    5:18.09.0~3-0~debian-stretch             amd64        Docker CLI: the open-source application container engine

ii  collectd                         5.7.1-1.1                                amd64        statistics collection and monitoring daemon
ii  collectd-core                    5.7.1-1.1                                amd64        statistics collection and monitoring daemon (core system)
root@asrv241 ~ # 

[haggy]

Im also seeing this. Stats below.

Ubuntu 18.04 server Docker version 19.03.5, build 633a0ea838 collectd 5.7.2.git

[blackandred]

Same issue there.

[sunil-samtani]

Same issue here and the collectd stats seem to be missing for each warning message

[CM000n]

I have the same problem here under Debian 10 and collectd 5.8.1-1.3 It seems that I don't get any blkio stats of the containers.

After some trial and error it seems to me that the error is caused by a wrong blkio_stats value. For me blkio_stats probably returns a value of 6 instead of 5, but unfortunately I have no idea what this means or how to solve it:

[thinkingerrol]

I have the same problem (lots of these warnings polluting the logs by repetition) on a host with Ubuntu 16.04 and docker-ce/xenial,now 5:20.10.5~3-0~ubuntu-xenial whereby the docker-collectd-plugin is running within my monitoring_container based on Ubuntu 20.04 : [2021-04-07 15:22:27] [warning] Unexpected number of blkio stats for container 026d72a/monitored_container: 5! This problem goes away if I run the same monitoring_container and monitored_container on a host with Ubuntu 20.04 and docker.io/focal-security,now 19.03.8-0ubuntu1.20.04.1.

Not sure whether the pesky warnings are caused by the different host OS version, the different choice of Docker distribution, or maybe my docker-ce is outdated or simply because the fact that my monitoring_container uses the Docker.io distro (packaged by Canonical) makes it incompatible with hosts which use the Docker CE distro (packaged by Docker Inc)