Mem usage doesn't match `docker stats -a`

[vgoklani]

Hey there,

I'm running the latest version of ctop, and when comparing the mem usage stats between docker stats -a, I've noticed that the values are all larger. Which app is correct, and why are the values larger?

Thanks!

[wlmirand]

I believe its a ctop issue. For one container ctop says it uses 3GB of ram, but docker stats -a reports less than 900mb. Since the overall memory usage checked using htop on my host machine says 2.2gb, I do believe docker stats is correct.

[bbsixzz]

Yes I can confirm, docker stats shows this:

CONTAINER ID   NAME        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
ca51d2673640   deluge      0.47%     307.7MiB / 15.51GiB   1.94%     0B / 0B          82.6MB / 7.8GB    14

Ctop shows this:

▶ deluge ca51d267364 0% 5G / 16G 0B / 0B 0B / 0B 14 11 hours

[DMW007]

I noticed the same: ctop and docker stats were opened in two SSH sessions on a split screen to make sure that It seems that the memory usage is fetched from the Docker API here:

c.MemUsage = int64(stats.MemoryStats.Usage - stats.MemoryStats.Stats.Cache)

First I thought that this calculation would explain the difference, but the docker stats API seems to do the same:

On Linux, the Docker CLI reports memory usage by subtracting cache usage from the total memory usage. The API does not perform such a calculation but rather provides the total memory usage and the amount from the cache so that clients can use the data as needed. The cache usage is defined as the value of total_inactive_file field in the memory.stat file on cgroup v1 hosts.

I tried to query the Docker API directly for the example container:

$ curl --silent -XGET --unix-socket /run/docker.sock http://localhost/containers/5448a42fa234/stats?stream=false | jq .memory_stats
{
  "usage": 531767296,
  "stats": {
    "active_anon": 11137024,
    "active_file": 229097472,
    "anon": 34324480,
    "anon_thp": 0,
    "file": 458465280,
    "file_dirty": 0,
    "file_mapped": 13746176,
    "file_writeback": 0,
    "inactive_anon": 24154112,
    "inactive_file": 229367808,
    "kernel_stack": 311296,
    "pgactivate": 85402,
    "pgdeactivate": 125321,
    "pgfault": 9911151,
    "pglazyfree": 9997,
    "pglazyfreed": 0,
    "pgmajfault": 34183,
    "pgrefill": 129975,
    "pgscan": 381701,
    "pgsteal": 239245,
    "shmem": 0,
    "slab": 37055848,
    "slab_reclaimable": 36716704,
    "slab_unreclaimable": 339144,
    "sock": 0,
    "thp_collapse_alloc": 0,
    "thp_fault_alloc": 13,
    "unevictable": 0,
    "workingset_activate": 0,
    "workingset_nodereclaim": 0,
    "workingset_refault": 0
  },
  "limit": 536870912
}
^C

This shows that .memory_stats.usage equals to the value displayed in ctop. We have 507,1 MiB from the API, but just because the value has changed during my research. In ctop, it now shows me 507 M too, instead of 509 M from the previous screenshot. If we calculate usage - inactive_file, this equals to 288,4 MiB and matches the docker stats value:

The Docker Documentation notes that the inactive_file field was defines as cache up to Docker 19.03. My machine runs Docker 24.0.6. Since the latest release of ctop is ~1.5 years old, it may be tested on an older Docker version and the used library could return 0 for the attribute, which doesn't exist any more. Further debugging would be required to find out what's exactly happening here and to be sure if this is a problem of ctop or the used library for fetching Dockers data.

Since it seems to be unmaintained, it probably won't get fixed in this project.

Update: Calling /stats with ?stream=false query parameter makes more sense here to get just a single snapshot instead of automated frequent updates.