search

litixsoft / karma-mocha-reporter

Karma reporter plugin with mocha style logging.
Other
197 stars 45 forks source link

Error sources occasionally being dropped from karma-mocha-reporter summary #99

Open rsimha opened 7 years ago

rsimha commented 7 years ago

We on ampproject/amphtml recently started using karma-mocha-reporter as the reporter for our tests on Travis-CI. I'm noticing that some of the time, the failures reported by karma-mocha-reporter are not accompanied by error sources. For example, see https://travis-ci.org/ampproject/amphtml/jobs/256983271

Any idea what's going on here? Is this a bug in how errors are being generated in the tests? Or is the reporter dropping them? Note that the progress indicator being used is super-dots, which does not print errors inline.

FYI, the source code for the failing tests in the example above are here, here, and here.

4kochi commented 7 years ago

Well, this issue is hard to track down. I ran the test locally and there were some errors. And the error sources are all listed. There were cases in the past were the error stack was not correctly reported to the reporter either by karma or the browsers. But was nothing what the reporter could have done.

rsimha commented 7 years ago

@4kochi, thanks. I'll ping this bug if we see this again.

rsimha commented 7 years ago

@4kochi Here's an update.

I encountered several more instances of unexpected error reports from karma-mocha-reporter. In some cases, it reported several alleged failures even though only one test failed. It's possible that this has to do with the fact that we're triggering tests on Travis and running them on Sauce Labs browsers. Either way, I thought you might find this useful to look at.

Here's a bug that describes what we're seeing: https://github.com/ampproject/amphtml/issues/11491 (I had to go use a different reporter for our tests that run on Sauce Labs.)

If you look at the logs in https://travis-ci.org/ampproject/amphtml/jobs/281179276#L694, you'll see that there was only one test failure, and yet, karma-mocha-reporter lists dozens of supposedly failing tests.