search

karma-runner / karma-coverage

A Karma plugin. Generate code coverage.
MIT License
770 stars 247 forks source link

Coverage timeouts and performance problems #221

Open mramato opened 8 years ago

mramato commented 8 years ago

I'm a maintainer on the Cesium project and we've recently switched our 200,000 line JavaScript code base over to use karma (over 7200 jasmine unit tests). Besides an issue with binary files that we submitted a PR for (which has since been merged). everything went off without a hitch and works phenomenally well. The entire test suite runs in less than a minute. Thank you for such a great project.

That being said, we are now trying to switch over code coverage from an ancient version of JSCoverage to istanbul via karma-coverage and running into a major performance hurdle. Our system takes about 90 seconds total to instrument and cover. However, running the same tests via karma-coverage goes on for several minutes and ultimately fails. I had to jack up browserNoActivityTimeout and browserDisconnectTimeout just to try and get it to run to completion (and that has only happened once so far and took 10 minutes, the results are not repeatable). A few seemingly innocuous tests seem to take multiple minutes to complete on their own (but execute in milliseconds when not run under coverage). I'm not sure it's any particular test causing issues though as commenting out "problem" tests still cause the whole thing to run slowly.

I've tried multiple launchers and browsers in case that was the problem (we usually use Chrome by default), but all browsers exhibited the same performance issue.

While I don't expect anyone to magically solve my problem, I am hoping that someone can provide tips for tracking down whatever the root cause of the problem may be. Perhaps there are some options I can tune or extra logging commands I can run to figure out what is taking so long.

Here's the relevant configuration we are using:

https://github.com/AnalyticalGraphicsInc/cesium/compare/coverage https://github.com/AnalyticalGraphicsInc/cesium/blob/coverage/Specs/karma-main.js https://github.com/AnalyticalGraphicsInc/cesium/blob/coverage/Specs/karma.conf.js

Any help would be greatly appreciated, and thanks again for a great project.

As a full disclaimer, I asked this question on the mailing list 3 weeks ago and received 0 replies, I'm hoping that I was just asking in the wrong place and that posting an issue here is okay. Thanks again.

dignifiedquire commented 8 years ago

Hey, sorry for not replying on the mailing list. The main issue is that I personally have very little understanding of istanbul and so are not really a big help here. From what it sounds like though the bottleneck is probably on istanbuls side rather than on karma-coverage as that is just grabbing that and reporting on it. So tagging @gotwarlost maybe he can help out better here, or give some insights.

mramato commented 8 years ago

Thanks for the reply. I forgot to mention that I actually opened https://github.com/gotwarlost/istanbul/issues/556 three weeks which didn't get any replies either. I feel spammy having opened multiple issues for the same thing, but I really wasn't sure which project was the root of the problem (and have been pulling my hair out trying to debug myself) So I'm just grateful for any sort of answer. Hopefully @gotwarlost can help point me in the right direction. Feel free to close this if you feel it's definitely an istanbul issue and I can always re-open if we find out otherwise.

Thanks again.