search

litixsoft / karma-mocha-reporter

Karma reporter plugin with mocha style logging.
Other
197 stars 45 forks source link

concurrent runs screw up log report #86

Open CendioOssman opened 8 years ago

CendioOssman commented 8 years ago

Testing multiple browsers at the same time confuses the reporting somehow and you can't really see which instance failed or succeeded, or where the output belongs.

Examples:

https://travis-ci.org/CendioOssman/noVNC/jobs/179805509 https://travis-ci.org/CendioOssman/noVNC/jobs/179770987

4kochi commented 8 years ago

I am not sure if the output is from the reporter. Actually it supports concurrent browsers. Could it be that the output is from some console.log() statements?

CendioOssman commented 8 years ago

Yes, it is. However I would expect that output to be either at the start or bottom, or at the test that generates the output. Right now it is very random.

More concerning is that:

a) there are tests that are marked as failed (red, and x), yet the entire run is considered a success b) you cannot see the test results for each platform. So failures might just be for one platform, and a success might be skipped for others.

4kochi commented 8 years ago

Regarding the console.log this seems to be the same problem as in #78. There is also an open issue in the karma repo (https://github.com/karma-runner/karma/issues/2342) which means this could be a general in how the reporters are supposed to use the logging API of karma. I will keep an eye on this

4kochi commented 8 years ago

The other problem with the different platforms is hard to track down for me. Might a problem which only occurs when using custom saucelab launchers. I will try to reproduce it.

wesleycho commented 7 years ago

This is a problem in karma itself IMO - karma is coupling the defining of default reporters with logging wired in, so reporters that are using the base reporter decorator are logging by default, which is probably not the intention. I have a proposed solution (https://github.com/karma-runner/karma/issues/2342#issuecomment-267064756) that I am leaning towards, and it would avoid needing to change anything in this library itself.

4kochi commented 7 years ago

Thanks for the information @wesleycho