aleks-mariusz
commented
5 years ago any updates on this ? there doesn't appear to have been any new releases to try (at least not since july 23rd when 1.9.1. was released).
Open aleks-mariusz opened 5 years ago
aleks-mariusz
commented
5 years ago any updates on this ? there doesn't appear to have been any new releases to try (at least not since july 23rd when 1.9.1. was released).
bboreham
commented
5 years ago Currently, Scope reports RSS for the process, and Docker's "Usage" stat for the container. I see that in https://github.com/docker/cli/pull/80 Docker's command-line tool changed to print a different stat - "Usage" minus "page cache".
To check, you can look at the contents of /proc/<pid>/status on the host and see which numbers match what appears on the screen.
Broadly, memory consumption is a complicated subject and when you reduce it to a single figure you will always lose something. But I agree a GUI tool like Scope should at least look consistent.
If this is the problem then this is a duplicate of #1133.
aleks-mariusz
commented
5 years ago this issue looks definitely related, but whether they are duplicates of each other hard to say.. that one issues has the process as larger than the container, in my situation the container is larger than the process.
so i took a look at the /proc/
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1000 4205 3.5 15.1 3870748 2456872 ? Ssl Oct12 2125:19 /usr/lib/jvm/default-jvm/jre/bin/java ... org.elasticsearch.bootstrap.Elasticsearch
1000 5401 3.7 15.0 3856684 2446464 ? Ssl Nov02 1086:17 /usr/lib/jvm/default-jvm/jre/bin/java ... org.elasticsearch.bootstrap.Elasticsearchhere we see that VSZ is ~1.5x times that of RSS (and yet scope reports it as the RSS size). VSZ column corresponds nicely with the VmSize line from /proc/
however, the container information (which i infer you're saying it comes from docker stats output) shows:
# docker stats --no-stream $(docker ps | awk '{if(NR>1) print $NF}')|grep es-data
k8s_es-data_es-data-1_monitoring_820e2a87-ded1-11e8-a4bb-5254004c9f89_0 12.59% 2.326 GiB / 15.51 GiB 15.00% 0 B / 0 B 10.2 MB / 107 GB 49
k8s_es-data_es-data-2_monitoring_663cb8a1-ce30-11e8-a4bb-5254004c9f89_0 0.24% 2.337 GiB / 15.51 GiB 15.06% 0 B / 0 B 120 MB / 220 GB 52Hence i am unclear still why the container size according to scope, is on the order of 4gb (each, for these two containers in my sample above), whereas docker stats says they're only 2gb each..
is scope obtaining its figures from something different than docker stats output ?
i have a lab cluster, running kubernetes v1.11.3 which works fine but when using scope it seems to report memory usage of a few select containers a lot more than the processes within use. asked on slack and was told it gets the info from docker stats. i checked docker stats but it too reports less for the container than what scope reports the container uses.
notice the es(-data-N) containers, are using almost 4gb of memory where as the process within are only using just over two.
the output of docker stats:
What you expected to happen?
container memory usage is in line with process it contains
What happened?
memory used by container reported looks almost twice that of what process it contains is
How to reproduce it?
run elasticsearch with -Xm{s,x}=2048m
Anything else we need to know?
baremetal kubernetes 1.11.3, configured for HA using kubeadm, using scope v1.9.1
Versions:
Logs: