Large rdm memory consumption (100G+) and Emacs lockup which maybe related to diagnostics

[JohnC32]

Please mark appropriate

• [x ] rtags (rdm/rc/rp)
• Emacs Lisp
  • [ ] ac-rtags.el
  • [ ] company-rtags.el
  • [ ] helm-rtags.el
  • [ ] flycheck-rtags.el
  • [ ] ivy-rtags.el
  • [x ] rtags.el

Problem description

I'm using the RTags commit 959e00 as of Dec-19-2017 and have enabled diagnostics, (setq rtags-autostart-diagnostics t). I'm using it on a medium size code base (~25000 *.cpp files plus many more headers). The processing of the files takes hours on a 12-core system. When processing is done, I see Emacs lock up. I believe this is related to rtags-diagnostics. Looking at the process tree for Emacs, I see:

rc --socket-file=/path/to/rdm_socket -m --elisp

I also see rdm consuming lots of CPU time and significant amount of memory (100G+).

Note, when I pass the compile commands to rdm, they include -Weverything, so there's going to be significant number of warnings per file.

Does the diagnostics capability in rtags.el apply to only the files loaded in Emacs or all files indexed by rdm? If it is all files, then this could explain the memory consumption.

The emacs hang, seems to be due to the processing of the diagnostics output. I notice that the hang occurs when another process is spawned by emacs, e.g. M-x man TOPIC while rtags-diagnostics is happening causes the hang and Emacs CPU usage to hit 100%.

Expected behavior

It would be nice to activate diagnostics and push the compiler warning to maximum (-Weverything), yet only report diagnostics on files loaded into Emacs so that we don't have scaling issues.

Environment

• Your operating system: Debian 8

• LLVM/Clang version: 5

[JohnC32]

I removed the -Weverything and still see issues. I just ran in a terminal:

rc --socket-file=/path/to/rdm_socket -m -elisp > m.log

and it is effectively hanging. Perhaps if I waited a very long time it would finish? The file it's creating is large. I ^C the command after 30 minutes with a m.log size of 500M.

[JohnC32]

It seems the diagnostics capability (and hence code completion) is not usable on medium to large size projects due to the scaling problems. It does work quite well on smaller projects. On larger projects, one can disable the diagnostics to prevent the scaling problems by defining the following after the (require 'rtags):

  (defun rtags-diagnostics (&optional restart nodirty)
    (interactive "P")
    (message "rtags-diagnostics has scalability issues and has been disabled.")

Also don't activate completions because this uses diagnostics.

[Nephyrin]

Part of the problem appears to be header files

Monitoring messages being delievered to emacs from rdm, it seems to behave well until a reparse of any header happens, at which point insane amounts of data is offloaded, in huge blocks:

$ rc -m --elisp | while IFS= read -d$'\n' -r line; do echo "$line" | wc; done
      1  637920 6176384
      1       5      32
      1  637848 6175012
      1       5      32
      1  637860 6175552
      1       5      32
      1  637860 6174760
      1       5      32
      1  637908 6175306
      1       5      32
      1  637908 6176064
      1       5      32

When rdm is idle, just doing:

rc -m > /tmp/rtags.log &
rc -V some_header.h

Shows 19 files get reparsed, but /tmp/rtags.log now contains 68M of warnings, from all 62,000 files in the project.

As for autocomplete, I've been using irony-mode which uses libclang but manages near-zero latency, and can be hooked up to read compilation flags from rtags. It also has a diagnostics/flycheck mode, but that seems well inferior to rtags diagnostics.

[Andersbakken]

I debugged this a little and found two relatively bad bugs.

4c16e1fb3aee88c6f20b23ae2a6a08032e09478c fixes an issue where if you ever had more than one indexable buffer visible in a frame rdm wouldn't be told about either of them.

06d8be3c76cc3a227606a8e11578fa8a3f3bb2f0 fixes an issue where rdm erroneously didn't discard diagnostics correctly. I believe this should improve matters drastically.

[Nephyrin]

This seems to be much improved for me - Diagnostics is snappy again, and I no longer see emacs chewing 100% when re-parsing is occurring.

(Thanks yet again for the awesome tool!)

[JohnC32]

Thanks - I've been using the updates for a bit and generally things are much better. There have been a couple stalls, but I'm not sure if it is related to diagnostics or not. I'll will continue to try to see where the stalls are coming from and report back if I can narrow down the issue.

As an aside, when indexing with -Weverything (minus a couple warnings), I see RTags take about 15% more time to create the index vs when compiling with -Wall (plus a couple warnings). This is on a medium size project of 25K files (203 minutes vs 177 minutes). Our code is fairly clean to the -Wall, but not clean to the -Weverything. I wonder if the extra time is coming from asking clang/llvm to do more diagnostic analysis or if the extra messages from clang/llvm are slowing down rtags.

[JohnC32]

I've been using RTags with diagnostics and completions active on a medium size project and it does well. rdm memory consumption is reasonable hovering around 2.5GB RES. There are slowdowns, but I think they may be unrelated to the diagnostics. The slowdowns in Emacs seem to be related to when rdm is doing work (running rp's). The system has plenty of headroom and other applications are very responsive when rdm is doing work. Emacs seems to be slow due to what seems to be to be unnecessary interactions with rdm and rdm is slow to respond (perhaps a locking issue?).

If I'm looking at some code and run M-x man TOPIC it take a long time (much longer than when rtags is not running). Switching to the RTags Log buffer, I see in the RTags Log a lot of activity. I see --set-buffer-files is called many times, some of which are redundant? Perhaps, this could be optimized.

 **********************************
 1516294735: --set-buffers files: /path/to/header.hpp
 **********************************
 1516294740: --set-buffers files: /path/to/header.hpp
 **********************************
 1516294740: --set-buffers files: /path/to/header.hpp
 **********************************
 1516294741: --set-buffers files: /path/to/header.hpp
 **********************************
 1516294741: --set-buffers files: ;
 **********************************
 1516294741: /path/to/rc --socket-file=/path/to/rdm_socket --silent-query --silent -b -z --verify-version=124 --set-buffers ;

[JohnC32]

It seems the --set-buffers is being used by rdm to filter diagnostics. Is there any reason to call --set-buffers without files? I assume it would be good to avoid the redundant calls to --set-buffers which could be done by caching in rtags.el what was last sent to rdm. This would reduce the traffic to rdm.

The other question / possibility is that the context switching rdm is doing while it is running rp is not optimal as the slowdown occurs only when rp is running.

[JohnC32]

One other observation: at times I see rdm go to 98 to 100% constant CPU for a while. When this happens emacs locks up. Usually this happens when I ask to do an operation such as finding references. The only way to recover is to kill rdm.

Would you know off hand how to build rdm in release mode, but with symbols so we can get crash dumps (what cmake configuration should we use and perhaps this should be the default)?ks

Thanks

[Nephyrin]

A few observations from using rtags recently.

On large files, diagnostics still hangs emacs when being parsed, usually for ~0.5 - ~3s. I accidentally found a good repro: Find a complex file, run rtags-preprocess-file, replace the file with its preprocessed self to get diagnostics (trying to resolve an odd nested-include error). The now much-larger file triggers pretty much all the bad diagnostics edgecases.

It seems like diagnostics should yield every N lines of diagnostics parsed so it's not hanging input for more than a few ms at a time, as any blocking while typing is very jarring, even the more-common half-second pauses I see on more real-world files.

Additionally, rtags seems to do blocking communication with rdm. If rdm is busy processing many files, it can take a second or so to respond, causing stutter.

Repro: Send SIGSTOP to rdm while in a diagnostics-having buffer. After a moment, some blocking rdm call will hang emacs. May be related to rtags-track-container?

[Andersbakken]

Hi John

Thanks. I'll see if I can repro and try to do figure out a solution. Did the file you used have a lot of warnings etc?

Anders

On Thu, Feb 1, 2018 at 4:17 PM, John Schoenick notifications@github.com wrote:

A few observations from using rtags recently.

On large files, diagnostics still hangs emacs when being parsed, usually for ~0.5 - ~3s. I accidentally found a good repro: Find a complex file, run rtags-preprocess-file, replace the file with its preprocessed self to get diagnostics (trying to resolve an odd nested-include error). The now much-larger file triggers pretty much all the bad diagnostics edgecases.

It seems like diagnostics should yield every N lines of diagnostics parsed so it's not hanging input for more than a few ms at a time, as any blocking while typing is very jarring, even the more-common half-second pauses I see on more real-world files.

Additionally, rtags seems to do blocking communication with rdm. If rdm is busy processing many files, it can take a second or so to respond, causing stutter.

Repro: Send SIGSTOP to rdm while in a diagnostics-having buffer. After a moment, some blocking rdm call will hang emacs. May be related to rtags-track-container?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Andersbakken/rtags/issues/1131#issuecomment-362445349, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdSrCTimUwEyc_kiNBWCCULwzR0Eecks5tQlQIgaJpZM4RTvzu .

[Thaodan]

Using any larger project will produce many warnings. For example when tring to open a file from the mozilla codebase. I can count the minutes that rtags shews my 32GB ram.

[Nephyrin]

Possibly related to this, I often see emacs hangs when rdm is very busy. I have a large project which is averaging upwards of ~10s per file for indexing according to rdm stats. Using toggle-debug-on-break and interrupting the hangs, the culprits are usually mundane calls like rc --set-buffers and container tracking. Is rdm maybe only responding to these queries as it's existing jobs complete?

Let me know if there are any debug steps I can try to narrow this down.

Edit: I just noticed this is a continuation of a much older incarnation of the same issue I responded to earlier, ha! FWIW this only recently (last ~6 months) started being an issue, so I think something improved and then regressed again since the earlier discussion.

[Nephyrin]

I've split out the current responsiveness issue I can reproduce into #1299