Docker reporting significantly different do what docker stats reports

[raqua]

I am not sure if this is a bug or some other interpretation of the data, but it seems that there are multiple incorrect values regarding my docker containers. Please see this attached screenshot: On top is output of docker stats.

Issues I have noticed:

1. Glances shows about double of docker container CPU usage. Looking at the overal usage shown by Glances, it looks like this is not correct (I am running on 4c/8t system).
2. Containers memory usage reported in Glacnes is too high, maybe includes also caches? Again, looking at the overall RAM usage, it looks like container memory usage is not reported correctly and docker stats is right.
3. Zoneminder container is receiving streams of 3 webcams and as you can see, eth0 and docker0 interfaces in Glances clearly show traffic flow, but in the container section, there is no Rx/Tx reported.
4. I believe that there are also some disk writes in Zoneminder container (includes MariaDB) which are also not reported in docker section of Glances.

I do not have lsb_release, but I am running fully up to date Arch linux.

# glances --version
Glances v3.1.3 with psutil v5.6.7

# uname -a 
Linux mars 5.4.5-arch1-1 #1 SMP PREEMPT Wed, 18 Dec 2019 19:48:51 +0000 x86_64 GNU/Linux

[nicolargo]

Strange behavor... What is your python Docker lib version ?

On my side:

$ python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> docker.__version__
'4.1.0'

[nicolargo]

1) For the CPU, the grab frequency is configure to 3 seconds in Glances and less than 1 second in docker stats. It can explain the difference. I have a look and i use exactly the same algo to compute the CPU% (see https://github.com/TomasTomecek/sen/blob/master/sen/util.py#L160-L171 and https://github.com/nicolargo/glances/blob/v3.1.3/glances/plugins/glances_docker.py#L301-L304) 2) I have the same value in Glances / docker stats concerning the memory 3) The Rx/Tx do not change also in the docker stats command... 4) Glances shows some Disk IO on my computer

[raqua]

My docker lib:

$ python
Python 3.8.1 (default, Dec 21 2019, 20:57:38) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker 
>>> docker.__version__
'4.1.0'

Could this be python 3 related? I am not sure which version is used to run Glances on my system. Any idea how do I check that?

1. I am not sure this is 3s vs. 1 second issue. By the way, I run Glances with 1 second interval. But this is not some one time difference, I get this double CPU usage reported all the time consistently.
2. Well ... can I do something to help troubleshoot why it behaves differently on my system ?
3. I think in Docker stats this is cumulative value, it changes on my system, but rather slowly.
4. It does show on mine now as well

[attzonko]

Any update on this I am seeing this exact same wonky behavior. I am using glances in a docker container. And the memory values are vastly different 549MB from docker stats vs 21GB reported for the same container in Glances.

UPDATE: Something of note on the memory side is that I see the same wrong number (~21GB) also being reported by cadvisor.

Are you using the Docker API to get the value of container memory usage in bytes?

[attzonko]

I think the issue is with whcih value of stats you use from the docker python library. I just confirmed that getting the stats from it, the 'usage' returned is ~21GB. This was done inside the glances container using:

$ docker exec -ti glances /bin/sh
# python
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> import docker
>>> client = docker.DockerClient(base_url='unix:///var/run/docker.sock')
>>> for i in client.containers.list():
...     if i.name == 'plexms':
...         pprint(i.stats(stream=False))
...

...
u'memory_stats': {u'limit': 33375371264,
                   u'max_usage': 26938580992,
                   u'stats': {u'active_anon': 76652544,
                              u'active_file': 212492288,
                              u'cache': 21908553728,
                              u'dirty': 405504,
                              u'hierarchical_memory_limit': 9223372036854771712,
                              u'hierarchical_memsw_limit': 0,
                              u'inactive_anon': 72044544,
                              u'inactive_file': 21695922176,
                              u'mapped_file': 41361408,
                              u'pgfault': 28369209,
                              u'pgmajfault': 462,
                              u'pgpgin': 61920639,
                              u'pgpgout': 56535328,
                              u'rss': 148758528,
                              u'rss_huge': 0,
                              u'total_active_anon': 76652544,
                              u'total_active_file': 212492288,
                              u'total_cache': 21908553728,
                              u'total_dirty': 405504,
                              u'total_inactive_anon': 72044544,
                              u'total_inactive_file': 21695922176,
                              u'total_mapped_file': 41361408,
                              u'total_pgfault': 28369209,
                              u'total_pgmajfault': 462,
                              u'total_pgpgin': 61920639,
                              u'total_pgpgout': 56535328,
                              u'total_rss': 148758528,
                              u'total_rss_huge': 0,
                              u'total_unevictable': 0,
                              u'total_writeback': 405504,
                              u'unevictable': 0,
                              u'writeback': 405504},
                   u'usage': 22485102592},
...

[attzonko]

If I had to guess docker stats reports the usage - cache which in my example above is about 549MB. I do believe that is a better metric to report.