Memory leak even without activity

ymartin59 commented 5 years ago

Output of awesome --version on Debian Testing

awesome --version awesome v4.3 (Too long) • Compiled against Lua 5.3.3 (running with Lua 5.3) • D-Bus support: ✔ • execinfo support: ✔ • xcb-randr version: 1.6 • LGI version: 0.9.2

How to reproduce the issue:

I start a small set of applications like Evolution, Pidgin, Emacs, Firefox, Google Chrome and keep them open. When system is locked for hours, awesome memory consumption increases a lot - even without activity - maybe except some applications' notifications in tray.

Actual result:

ps -eo comm,rss | grep awesome
awesome         70880

ps -eo comm,rss | grep awesome
awesome         1741096

Note that calling collectgarbage() through the lua prompt frees no memory (after some cpu consumption):

ps -eo comm,rss | grep awesome
awesome         1749104

Expected result:

Memory use stays approximately the same.

psychon commented 5 years ago

Does this happen with the default config or with your own config?
In case of your own config: Could you provide your config file(s)?
Does this only happen when the system is locked, but not when it is unlocked? (You seem to be implying so)
How do you lock your system?
What is the time difference between the ps calls you show?
Since you say that collectgarbage() does not help: What does awesome-client 'return collectgarbage("count")' say? (My AwesomeWM is at 43780 RSS according to ps and at 14342.1 according to collectgarbage, so... both count KiB?)
When did this start to happen? Did you just not notice this earlier, did this not happen in older versions, or are you just not using AwesomeWM for long? :-)

ymartin59 commented 5 years ago

I run AwesomeWM and only it since 2014, without significant changes in usage - except I now use (intensively) VMware Workstation instead of VMware Player

As far as I guess, memory leak happens since Debian package upgrade to 4.3 or switch to VMware Workstation

/var/log/dpkg.log.4.gz:2019-03-16 07:47:36 upgrade awesome:amd64 4.2-3 4.3-4
/var/log/dpkg.log.4.gz:2019-03-16 07:47:37 upgrade awesome-extra:all 2017110501 2019021001

Here is my configuration - I did no change when updating from 4.2 to 4.3: rc.lua.zip
I lock with gnome-screen-saver Debian service and command gnome-screensaver-command --lock
There is chance leak also happen unlocked with activity, I have to monitor process today

Previous measure was at 2019.06.17 ~7:50

ps -eo comm,rss | grep awesome
awesome         70880

And today, without restart, after 8 hours with activity and ~56 hours locked

$ date; awesome-client 'return collectgarbage("count")'
Thu 20 Jun 2019 10:08:37 AM CEST
   double 882270
$ date; ps -eo comm,rss | grep awesome
Thu 20 Jun 2019 10:08:46 AM CEST
awesome         1120920
$ date; awesome-client 'return collectgarbage("count")'
Thu 20 Jun 2019 10:10:17 AM CEST
   double 883352

I will use memleak from bcc https://github.com/iovisor/bcc with hope to grab a stack trace

ymartin59 commented 5 years ago

Few hours later with activity:

$ date; awesome-client 'return collectgarbage("count")'
Thu 20 Jun 2019 04:14:09 PM CEST
   double 995527

ymartin59 commented 5 years ago

Looks like leak is continuous:

date; awesome-client 'return collectgarbage("count")'
Fri 21 Jun 2019 09:10:40 AM CEST
   double 1.22301e+06

psychon commented 5 years ago

Random observations:

$ date; awesome-client 'return collectgarbage("count")'
Thu 20 Jun 2019 10:08:37 AM CEST
   double 882270
$ date; ps -eo comm,rss | grep awesome
Thu 20 Jun 2019 10:08:46 AM CEST
awesome         1120920

Awesome'S RSS is about 1094 MiB. Lua says that about 860 MiB of memory are allocated in Lua. Thus, this seems to be something quite-lua-only. Since you say that collectgarbage() does not help... well, I really have no idea, sorry.

Just for completeness: How about awesome-client 'local a = collectgarbage("count") collectgarbage() return a, collectgarbage("count")'? Does this at least show some decrease in Lua's claimed memory usage?

Here is my configuration - I did no change when updating from 4.2 to 4.3:

Looks like pretty much the default config with no special magic. The only magic (verticaltag) is unused.

/var/log/dpkg.log.4.gz:2019-03-16 07:47:36 upgrade awesome:amd64 4.2-3 4.3-4

Well, that narrows it down to about 500 commits. :-/

(Also, no idea what kind of effect a screen saver should have on the WM...)

Elv13 commented 5 years ago

Well, that narrows it down to about 500 commits. :-/

And a lot of them authored by me... Oh well, I guess its my turn to show up here. @ymartin59 are you familiar with git-bisect? What is your distribution? I didn't notice more issues regarding memory usage and not many people reported issues with v4.3 regarding memory. Normal memory leak detector like ASAN or Valgrind wont be of much use for Lua leaks. If you could narrow this down to a smaller commit range, I can review them more carefully.

ymartin59 commented 5 years ago

Thanks. Here are details:

$ awesome-client 'local a = collectgarbage("count") collectgarbage() return a, collectgarbage("count")'
   double 903518
   double 83153.8

When top line is

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                        
 2250 yma       20   0 1971448   1.7g  13584 S   0.0  10.9  29:11.04 awesome

If you confirm it is a Lua leaks, then it may be possible to use bcc/memleak as far as:

lua code is compiled with JIT
JIT compiler generates symbols (source file and line number...) with mapping file

I am used to diagnose Java native memory leak that way: https://github.com/jvm-profiling-tools/perf-map-agent. But I am not a Lua expert (yet), do you have answers concerning these two prerequisites?

psychon commented 5 years ago

double 903518 double 83153.8

Uhm. So, before the GC run, memory usage was about 900 MiB and afterwards it is down to 83 MiB? That's quite a difference!

So... something is just generating garbage in a way so that the GC cannot keep up?

JIT compiler generates symbols (source file and line number...) with mapping file

Sorry, but I doubt a lot that LuaJIT generates hints for profilers. Valgrind generates output like this: https://github.com/awesomeWM/awesome/issues/2288#issuecomment-395970221 The backtraces basically just say what kind of Lua object is being allocated. And perhaps provide some hints as to where that comes from. So, while this kind of information is certainly useful, it is not as helpful as you seem to be used to from Java.

ivan-kolmychek commented 5 years ago

(Message extracted into #2858).

psychon commented 5 years ago

@ivan-kolmychek Please open a new issue.

From a quick look at your config: Does the issue go away when you remove your wallpaper timer?

ivan-kolmychek commented 5 years ago

@psychon thanks, I will check that and if it doesn't go away then I will extract my previous message into the new issue.

ivan-kolmychek commented 5 years ago

@psychon it seems to not solve the issue, so I've created #2858, I've copied my original comment there so I'll remove it from here, so it doesn't interfere.

ymartin59 commented 5 years ago

I confirm awesome memory increases regularly. I run collectgarbage often to keep its virtual memory size low... but I have faced a X crash recently and used sysrq commands to reboot.

sylvestre commented 4 years ago

I am experiencing the same issue with 4.3-4. Enough pain that I am considering moving to another WM

On two systems:

% awesome-client 'local a = collectgarbage("count") collectgarbage() return a, collectgarbage("count")'

   double 455647
   double 173423

% awesome-client 'local a = collectgarbage("count") collectgarbage() return a, collectgarbage("count")'

   double 402436
   double 148272

psychon commented 4 years ago

@sylvestre usual work around is to start a timer that makes the garbage collector do some work. Something like gears.timer.start_new(600, function() collectgarbage("step", 1024) return true end). The step argument here means "dear GC, pretend that 1 MiB of memory got allocated just now". This works around the usual problems where LGI creates garbage in a way that Lua's GC cannot easily deal with.

Personally, I am not experiencing this (enough for it to be a problem?).

electron271 commented 2 years ago

I've been getting this issue, and is the reason I'm considering leaving awesome. I'll log on, and my memory usage will increase to 100% in about 1-3 minutes. I do not experience this issue on other window managers.

actionless commented 2 years ago

@electron271 it seems like your rc.lua file have some problems, try with default rc.lua from /etc/xdg/awesome/rc.lua

awesomeWM / awesome

Memory leak even without activity #2798