How to debug large projects?

[ryanjarvis]

First off, thanks for this tool. This has massive potential.

So I set up a very small test project with only two files and a requirejs module and everything works fine. The commands work, omnicomplete works and when I run the tern server manually with the verbose option, I see a logical request and response returned. The installation worked.

However, when I try to use it in my main projects, which has a handful of files, lots of requirejs modules and 3rdparty libs like AngularJS, every command returns a "timed out" message, omnicomplete fails to match the pattern (even on things defined in the same file) and the tern server only shows the request, not a response (even with a timeout set to 60 seconds)

What are some strategies I can do to debug the issues here?

[marijnh]

Does the node process for the server sit at 100%? If so, does it get anywhere when you let it run for a long period (say, a minute)? Can you create a zip of the project with which this happens, or is it proprietary code?

[ryanjarvis]

I am not seeing the node process do anything even with a timeout set to 60. No CPU usage. No memory bump.

Fortunately the main project is also a test project of sorts, just larger, so I can zip it up for you. How do I get it to you?

[marijnh]

You can mail it to marijnh@gmail.com

[ryanjarvis]

Sent. Thanks.

[marijnh]

Well, 650 files totalling 160k lines of JavaScript is not what I'd consider 'a handful of files'. At this scale Tern isn't expected to work anymore. Its complexity is not linear to code size, so though it's very fast on small codebases, and reasonably fast on medium ones, for huge codebases it'll take too much time to be useful.

There is currently no way to tell the requirejs plugin to ignore some modules, except with the rather crude override option, which you can set to a JSON structure that defines the type in your library (or "{}" to simply treat it as empty).

[marijnh]

With the latest Tern code it is now possible to set a dontLoad option in your .tern-project which (using and array of glob patterns like foo/*.js or node_modules/something/**) can be used to prevent the server from loading certain files. It's still not a very elegant solution, but might make Tern halfway useable for your project.

[paranormal]

Yeap, I have the same issue. A one big js file (700K), "timed out" message appears always.

[dhruvasagar]

Sometimes (I haven't been able to debug yet) it does throw 'timed out', but on subsequent tries it starts to work again, I believe the ternjs process gets terminated perhaps, though there are occasions when subsequent :TernDef commands keep throwing 'timed out' and every time a new ternjs process is started. I have to :qa out and come back in to fix it. I'll be giving the code a closer look.

[joeytwiddle]

I am also working on a large project, and whitelisted all the files with loadEagerly. I get the timed out message when I first try auto-completion, but if I give Tern time to do its processing, then on later queries it works pretty nicely.

One concern is, if I retry completion while Tern is still running, then additional node processes fire up in parallel. I don't know if they are sharing the work; I am worried that each one is repeating the same work!

Due to the size of my project, sometimes these processes end up filling all my RAM! (Fortunately I have earlyoom running so they get killed before bringing my system down.)

I wonder if some improvements can be made to prevent multiple servers from firing up in parallel.

[marijnh]

They are definitely repeating the same work (there's no multiprocessing in the Tern server). More intelligent process management, and handling of busy/stuck processes would be a great addition to Tern (probably best implemented as another layer between the current server and the client), but someone would need to find the time to design and implement it.