Diagnostics slow in large project

[suo]

Please mark appropriate

• [+] rtags (rdm/rc/rp)
• Emacs Lisp
  • [ ] ac-rtags.el
  • [ ] company-rtags.el
  • [ ] helm-rtags.el
  • [ ] flycheck-rtags.el
  • [ ] ivy-rtags.el
  • [ ] rtags.el

Problem description

Hello there!

I am exploring integrating RTags with Nuclide. So far it works quite well even in FB's very very large codebase. However, coming up with diagnostics after saving a buffer regularly takes 20+ seconds. For example:

[100%] 1/1 00:02:17 /this/is/a/fake/name.cpp in 24829ms. (409 syms, 399 symNames, 25 includes, 1 of 3829 files, symbols: 883 of 125150, 125946 cursors, 976337 bytes written) (24564/248/5ms) (dirty). (priority 1)
Jobs took 24.96s. We're using 113mb of memory.

Incremental diagnostic latency in ycmd or Nuclide's current libclang service (using the same compilation flags) is typically <1 second, so I wonder if I'm configuring rdm incorrectly here.

I see there was some previous discussion around slow diagnostic times (#719) but it doesn't look like there was a clear resolution.

I'm running rdm with:

rdm --no-filemanager --data-dir /data/users/$USER/rtags_data_dir

[suo]

Looking at the code, it seems that diagnostics jobs create the translation unit from scratch, as opposed to using clang_reparseTranslationUnit. Reparsing is used in completion though.

So I'm thinking one way to speed things up is:

1. Keep a registry of open buffers (I think this already exists)
2. Cache the translation units corresponding to those buffers in memory
3. Prefer to reparse the cached translation unit instead of building a new one from scratch.

Would you be amenable to a PR that does the above?

[Andersbakken]

We have had something like this before. The main obstacle is that the process that does clang_parseTranslationUnit is restarted for each file we index.

The main reason we do this is to prevent the server from growing into a gigantic hog in terms of memory usage and also because clang crashes the whole app with some regularity.

I have considered the following approaches:

• gather diagnostics from the completion thread as well. This seems like a win-win scenario. I started on this over here:

https://github.com/Andersbakken/rtags/tree/codecompletediagnostics

• serializing the translation units of certain

I have done some experiments with serializing the translation unit to disk and reopening it which I believe could work well.

I just revived that effort somewhat in my last commit. It's not working yet but I've started on it in b56ec32e

I'm a little hesitant to start doing the parsing of translation units inside of rdm but I'm definitely open to any PR that improves our performance.

regards

Anders

On Wed, Sep 6, 2017 at 2:29 PM, michaelsuo notifications@github.com wrote:

Looking at the code, it seems that diagnostics jobs create the translation unit from scratch, as opposed to using clang_reparseTranslationUnit. Reparsing is used in completion though.

So I'm thinking one way to speed things up is:

1. Keep a registry of open buffers (I think this already exists)
2. Cache the translation units corresponding to those buffers in memory
3. Prefer to reparse the cached translation unit instead of building a new one from scratch.

Would you be amenable to a PR that does the above?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Andersbakken/rtags/issues/1049#issuecomment-327617708, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdStccrAydlOy15D8oS8Lk4389tpatks5sfw6zgaJpZM4PN8yk .

[suo]

Good idea to reuse the completion thread to do diagnostics. I might pick up where you left off; what do you think still needs to be done?

[Andersbakken]

Hi Michael

Sorry about the delay. I actually finished up the work. I haven't tested it that carefully but strictly speaking we should now get diagnostics updates from the completion thread when completions are enabled.

Anders

On Thu, Sep 7, 2017 at 6:09 PM, michaelsuo notifications@github.com wrote:

Good idea to reuse the completion thread to do diagnostics. I might pick up where you left off; what do you think still needs to be done?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Andersbakken/rtags/issues/1049#issuecomment-327878710, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdSoDhzqWxZoJZvLVZVrJmBDZxShqpks5sgDCKgaJpZM4PN8yk .

[suo]

Anders,

Brilliance! I will try it out on our side and let you know how things are going.

[baolonglin]

I have one large project, after change one header file, it will trigger several files to be recompiled, update diagnostic will return more than 2,300,000 characters per each file (the project is not so clean, many warnings, the diagnostic for each file have duplicate information), in the end the emacs will very slow with 100% cpu usage, busy parsing diagnostics. But the server side is finished quite soon.

Looks like my scenario is different with michaelsuo's. Is it a known issue?

[dvzubarev]

@baolonglin I have the same issue. Is there an option to make recompilation of files to be lazy?

[Andersbakken]

This should have gotten vastly improved as of 06d8be3c76cc3a227606a8e11578fa8a3f3bb2f0

Which version are you on?

regards

Anders

On Fri, Jan 19, 2018 at 7:36 AM, dvzubarev notifications@github.com wrote:

@baolonglin https://github.com/baolonglin I have the same issue. Is there an option to make recompilation of files to be lazy?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Andersbakken/rtags/issues/1049#issuecomment-359000158, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEdSpimATDjviWgqfc18811JgLWdE8Oks5tMLaXgaJpZM4PN8yk .

[dvzubarev]

Thank you for your response. I've seen this commit just an hour ago and updated. Also I suppressed warnings from third-party libraries. Now I'm testing and it seems that the issue is solved for me. Thanks!