search

nicolargo / glances

Glances an Eye on your system. A top/htop alternative for GNU/Linux, BSD, Mac OS and Windows operating systems.
http://nicolargo.github.io/glances/
Other
26.97k stars 1.54k forks source link

Possible memory leak #2976

Closed janusn closed 3 weeks ago

janusn commented 1 month ago

Check the bug Before filling this bug report, please search if a similar issue already exists. In this case, just add a comment on this existing issue.

Describe the bug A clear and concise description of what the bug is.

The memory usage reported by docker stats keeps increasing on 3 separated instances running on 3 different machines.

To Reproduce Steps to reproduce the behavior:

  1. Run the following command as root: # docker compose up -d
  2. add a line to the crontab of the root: 
    * * * * * /usr/bin/docker stats --no-stream --format '{{.MemUsage}}' glances | cut -d '/' -f 1 >>/var/log/docker_stats/glances.log
  3. Monitor the file /var/log/docker_stats/glances.log

Expected behavior A clear and concise description of what you expected to happen.

I expect the memory logged in the glances.log to be stable over time.

Screenshots If applicable, add screenshots to help explain your problem.

Environement (please complete the following information)

Additional context Add any other context about the problem here.

Here is a sample of the content of the file glances.log after 2 days of running:

112.4MiB 
113.4MiB 
113.4MiB 
113.4MiB 
113.4MiB 
113.1MiB 
113MiB 
113.5MiB 
113.5MiB 
113.5MiB 
113.5MiB 
113.5MiB 
113.4MiB 
* snip … snip *
193.8MiB 
194MiB 
194MiB 
194MiB 
194.1MiB 
194.2MiB 
194.3MiB 
194.3MiB 
194.4MiB 
194.5MiB 
194.7MiB 
194.6MiB 
194.6MiB 
194.6MiB 
194.6MiB 
194.8MiB

the content of the compose.yaml:

---
#version: "3"
services:
  glances:
    image: nicolargo/glances:latest-full
    container_name: glances
    privileged: true
    deploy:
      resources:
        limits:
          memory: 512M
    pid: host
    network_mode: host
    environment:
      - "TZ=Europe/London"
      - "GLANCES_OPT=-C /config/glances.conf -w --password -u ha"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config:/config:ro
    secrets:
      - source: ha_password
        target: /root/.config/glances/ha.pwd
    restart: unless-stopped
secrets:
  ha_password:
    file: ./secrets/ha_password.txt

modification of glances.conf:

[gpu]
disable=True
[hddtemp]
disable=True
[graph]
generate_every=0
[diskio]
hide=loop.*,/dev/loop.*,sd*
[fs]
hide=/boot.*,/snap.*,/etc/*,_config/*

You can also pastebin:

nicolargo commented 1 month ago

Hi @janusn

in the Glances configuration file you have the following key:

[global]
# History size (maximum number of values)
# Default is 1200 values (~1h with the default refresh rate)
history_size=1200

So it's a "normal" behavior to have your memory increase for the first hour.

After that it should be stable.

janusn commented 1 month ago

I have been monitoring the memory consumption since your reply. There were a few set backs such as container recreated by watchtower. Nonetheless, here is what I have found.

The memory usage on 2 of 3 instances of the glances keep growing over 24 hours period but the other one does not. Here is a summary of memory usage of one of the growing instances:

73.51MiB (runtime 00:00 hours)
104.9MiB (runtime 01:00 hour)
105.3MiB (runtime 02:00 hours)
107.7MiB (runtime 03:00 hours)
108.3MiB (runtime 04:00 hours)
109.8MiB (runtime 05:00 hours)
110MiB (runtime 06:00 hours)
112.3MiB (runtime 07:00 hours)
113.9MiB (runtime 08:00 hours)
114.7MiB (runtime 09:00 hours)
115MiB (runtime 10:00 hours)
117.1MiB (runtime 11:00 hours)
116.1MiB (runtime 12:00 hours)
117MiB (runtime 13:00 hours)
117.3MiB (runtime 14:00 hours)
119.1MiB (runtime 15:00 hours)
120.7MiB (runtime 16:00 hours)
121.5MiB (runtime 17:00 hours)
121.1MiB (runtime 18:00 hours)
121.7MiB (runtime 19:00 hours)
122.2MiB (runtime 20:00 hours)
122.6MiB (runtime 21:00 hours)
124.8MiB (runtime 22:00 hours)
124.5MiB (runtime 23:00 hours)
126MiB  (runtime 24:00 hours)
126.2MiB (runtime 25:00 hours)
125.4MiB (runtime 26:00 hours)
126.3MiB (runtime 27:00 hours)
127.4MiB (runtime 28:00 hours)
127.6MiB (runtime 29:00 hours)
129.7MiB (runtime 30:00 hours)
130.4MiB (runtime 31:00 hours)
130.4MiB (runtime 32:00 hours)
132.2MiB  (runtime 33:00 hours)
nicolargo commented 1 month ago

On my configuration (running Glances in local outside a Docker container) and with a history_size=3.

When Glances start:

rss=64245760, vms=590016512, shared=17039360, text=3026944, lib=0, data=91455488, dirty=0, uss=51081216, pss=52712448, swap=0

One hour later:

rss=65421312, vms=590155776, shared=17039360, text=3026944, lib=0, data=91594752, dirty=0, uss=51511296, pss=53140480, swap=0

So memory increase around 1.4 MB (regarding 31 MB on your test).

I will make a long term test next week.

nicolargo commented 3 weeks ago

Also reproduced on my side (in console mode with default history_size=1200) during 30 hours but with a more limited leak:

image

Test with history_size=0 (in red):

image

nicolargo commented 3 weeks ago

Note for myself: https://pythonhosted.org/Pympler/muppy.html

nicolargo commented 3 weeks ago

Ok found it thanks to mem_top:

2024-10-31 08:40:16,583 -- INFO -- 
refs:
1519    <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386    <class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
870 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
870 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
870 <class 'dict'> {3119283: {'cpu_percent': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
435 <class 'list'> [{'cpu_percent': 12.2, 'num_threads': 1, 'name': 'python', 'memory_percent': 0.19223070159317707, 'g
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816    {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960    {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
36960    {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
18520    {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520    {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520    {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520    {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi

types:
11694    <class 'function'>
9483     <class 'dict'>
7183     <class 'tuple'>
2653     <class 'weakref'>
2237     <class 'list'>
2050     <class 'cell'>
2003     <class 'getset_descriptor'>
1574     <class 'type'>
1394     <class 'wrapper_descriptor'>
1339     <class 'builtin_function_or_method'>

===

2024-10-31 09:26:12,157 -- INFO -- 
refs:
1690    <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
1519    <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386    <class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 551316), 2.0), (datetime.datetime(2024, 10, 31, 8, 40, 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 551311), 7.1), (datetime.datetime(2024, 10, 31, 8, 40, 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 552876), 64.8), (datetime.datetime(2024, 10, 31, 8, 40,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 552878), 84.9), (datetime.datetime(2024, 10, 31, 8, 40,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 553628), 0.74), (datetime.datetime(2024, 10, 31, 8, 40,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 553631), 0.64), (datetime.datetime(2024, 10, 31, 8, 40,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 556058), 1.3), (datetime.datetime(2024, 10, 31, 8, 40, 

bytes:
73816    {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960    {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
36960    {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
36960    {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
36960    {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520    {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520    {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal

types:
11694    <class 'function'>
9905     <class 'dict'>
8377     <class 'tuple'>
3985     <class 'list'>
2651     <class 'weakref'>
2050     <class 'cell'>
2003     <class 'getset_descriptor'>
1574     <class 'type'>
1394     <class 'wrapper_descriptor'>
1338     <class 'builtin_function_or_method'>

===

AFTER a F5 (clear processlist cache)

2024-10-31 09:31:11,215 -- INFO -- 
refs:
1519    <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386    <class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 456881), 0.5), (datetime.datetime(2024, 10, 31, 8, 45, 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 456876), 2.4), (datetime.datetime(2024, 10, 31, 8, 45, 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 458499), 64.8), (datetime.datetime(2024, 10, 31, 8, 45,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 458501), 84.9), (datetime.datetime(2024, 10, 31, 8, 45,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 459220), 0.52), (datetime.datetime(2024, 10, 31, 8, 45,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 459222), 0.56), (datetime.datetime(2024, 10, 31, 8, 45,
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 461633), 1.3), (datetime.datetime(2024, 10, 31, 8, 45, 
1200    <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 461635), 0.4), (datetime.datetime(2024, 10, 31, 8, 45, 

bytes:
73816    {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960    {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
36960    {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
18520    {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520    {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520    {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520    {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi

types:
11694    <class 'function'>
9433     <class 'dict'>
8377     <class 'tuple'>
3552     <class 'list'>
2651     <class 'weakref'>
2050     <class 'cell'>
2003     <class 'getset_descriptor'>
1574     <class 'type'>
1394     <class 'wrapper_descriptor'>
1338     <class 'builtin_function_or_method'>
nicolargo commented 3 weeks ago

The issue is confirmed and is in the processes.py file / class GlancesProcesses / dict self.processlist_cache.

The size of this dict can be very large because the key is the PID, on Linux system the default value for this file, 32768, results in the same range of PIDs as on earlier kernels. On 32-bit platforms, 32768 is the maximum value for pid_max. On 64-bit systems, pid_max can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million)...

So we need to have a peace of code after the main loop of the update method to clean key/value when PID is no more existing:

nicolargo commented 3 weeks ago

Adding this code after the loop:

        # Remove non running process from the cache (avoid issue #2976)
        pids_running = [p['pid'] for p in processlist]
        pids_cached = [p for p in self.processlist_cache.keys()]
        for pid in pids_cached:
            if pid not in pids_running:                                                                               
                self.processlist_cache.pop(pid, None)

seems to solve the leak:

Running Glances with (history_size=0):

$ while true; do ./venv/bin/python -c 'import os; import time; import psutil; g = psutil.Process(3223060); print("{} {}".format(time.strftime("%Y-%m-%d %H:%M:%S"), g.memory_full_info()));'; sleep 300; done
2024-10-31 16:10:03 pfullmem(rss=70799360, vms=1403809792, shared=15728640, text=4096, lib=0, data=152330240, dirty=0, uss=60723200, pss=63950848, swap=0)
2024-10-31 16:15:03 pfullmem(rss=70557696, vms=1403809792, shared=15728640, text=4096, lib=0, data=152330240, dirty=0, uss=60678144, pss=63902720, swap=0)
2024-10-31 16:20:03 pfullmem(rss=72482816, vms=1405534208, shared=15728640, text=4096, lib=0, data=154054656, dirty=0, uss=62652416, pss=65876992, swap=0)
2024-10-31 16:25:03 pfullmem(rss=71524352, vms=1404329984, shared=15728640, text=4096, lib=0, data=152850432, dirty=0, uss=61501440, pss=64351232, swap=0)
2024-10-31 16:30:03 pfullmem(rss=71847936, vms=1404592128, shared=15728640, text=4096, lib=0, data=153120768, dirty=0, uss=61755392, pss=64605184, swap=0)

Also confirmed by mem_top:

2024-10-31 16:07:57,006 -- INFO -- 
refs:
1519    <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386    <class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
890 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
890 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
890 <class 'dict'> {16089: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splittabl
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
445 <class 'list'> [{'num_threads': 26, 'pid': 16089, 'memory_percent': 2.455672030884629, 'cpu_percent': 19.7, 'gids':
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816    {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960    {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
36960    {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
18520    {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520    {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520    {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520    {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi

types:
11701    <class 'function'>
10050    <class 'dict'>
7218     <class 'tuple'>
2693     <class 'weakref'>
2584     <class 'list'>
2090     <class 'cell'>
2003     <class 'getset_descriptor'>
1574     <class 'type'>
1514     <class 'builtin_function_or_method'>
1394     <class 'wrapper_descriptor'>

===

2024-10-31 16:34:09,533 -- INFO -- 
refs:
1519    <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386    <class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
926 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
926 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
926 <class 'dict'> {3223060: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
463 <class 'list'> [{'num_threads': 9, 'pid': 3223060, 'memory_percent': 0.21623942888930556, 'cpu_percent': 14.1, 'gid
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816    {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960    {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
36960    {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
36960    {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520    {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520    {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520    {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa

types:
11701    <class 'function'>
10147    <class 'dict'>
7218     <class 'tuple'>
2698     <class 'list'>
2693     <class 'weakref'>
2090     <class 'cell'>
2003     <class 'getset_descriptor'>
1574     <class 'type'>
1514     <class 'builtin_function_or_method'>
1394     <class 'wrapper_descriptor'>

2024-10-31 16:34:11,742 -- INFO -- 
refs:
1519    <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386    <class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
926 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
926 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
926 <class 'dict'> {2175857: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
463 <class 'list'> [{'num_threads': 22, 'pid': 2175857, 'memory_percent': 10.093652599903841, 'cpu_percent': 151.6, 'gi
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816    {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960    {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
36960    {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
36960    {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520    {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520    {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520    {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520    {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa

types:
11701    <class 'function'>
10148    <class 'dict'>
7218     <class 'tuple'>
2699     <class 'list'>
2693     <class 'weakref'>
2090     <class 'cell'>
2003     <class 'getset_descriptor'>
1574     <class 'type'>
1514     <class 'builtin_function_or_method'>
1394     <class 'wrapper_descriptor'>

had to make some extra-test before pushing but look ok.

nicolargo commented 3 weeks ago

Ok pushed on the develop branch.

After some test, il will release a version 4.2.1.

nicolargo commented 3 weeks ago

@janusn Glances 4.2.1 is available. You can upgrade your system.

janusn commented 3 weeks ago

@nicolargo

Update:

I checked again and I realized that the image tagged 4.2.1-full is different from the image I was running, tagged latest-full. Let me run the 4.2.1-full a couple day. I will report my result afterwards

$ sudo docker image ls
REPOSITORY                        TAG           IMAGE ID       CREATED         SIZE
nicolargo/glances                 4.2.1-full    a03e9d74b587   3 days ago      197MB
nicolargo/glances                 latest-full   d9f6a9532303   3 days ago      197MB

Original comment:

Thanks for the quick fix.

On one of the 3 containers, the memory usage is still growing albeit it is much slower than before.

11/2
05:17:03 65.92MiB (container restarted by watchtower)
06:00:03 70.41MiB
07:00:03 79.04MiB
08:00:03 96.26MiB
09:00:03 98.54MiB
10:00:03 97.65MiB
11:00:03 97.6MiB
12:00:04 98.7MiB
13:00:03 98.39MiB
14:00:03 99.37MiB
15:00:03 99.45MiB
16:00:03 99.29MiB
17:00:04 98.3MiB
18:00:03 99.8MiB
19:00:03 99.16MiB
20:00:03 100.5MiB
21:00:03 99.81MiB
22:00:03 99.62MiB
23:00:03 99.59MiB
11/3
00:00:04 99.93MiB
01:00:04 99.63MiB
02:00:03 99.88MiB
03:00:04 99.86MiB
04:00:03 100MiB
05:00:03 100.1MiB
06:00:03 101.7MiB
07:00:03 100.5MiB
08:00:03 100.7MiB
09:00:04 100.7MiB
10:00:03 101.4MiB
11:00:03 101.4MiB
12:00:04 101.4MiB
13:00:04 101.4MiB
14:00:04 101.4MiB
15:00:03 104.2MiB
16:00:03 104.3MiB
17:00:04 104.5MiB
18:00:04 104.4MiB
19:00:03 103.5MiB
20:00:02 104.7MiB
21:00:03 104.4MiB
22:00:03 104.4MiB
23:00:03 104.6MiB
11/4
00:00:03 103.5MiB
01:00:04 103.6MiB
02:00:03 105.2MiB
03:00:03 105.3MiB
04:00:03 105.1MiB
05:00:03 104.2MiB
06:00:03 105.3MiB
07:00:03 104.3MiB
08:01:04 104.2MiB
09:00:03 105.5MiB
10:00:04 104.5MiB
11:00:04 104.5MiB
12:00:03 105.7MiB
13:00:04 105.4MiB
14:00:03 105.6MiB
15:00:03 105.4MiB
16:00:03 106.2MiB
17:00:03 106.1MiB
18:00:03 105.7MiB
19:00:03 106.2MiB
20:00:03 106.2MiB
21:00:03 106.4MiB
22:00:03 106.4MiB
23:00:04 107.7MiB
11/5
00:00:03 108.1MiB
janusn commented 2 weeks ago

Unfortunately, 4.2.1-full still exhibits this memory leak. I have included logs from the three install starting from restarting of the containers. logs.zip