ConsolidateAll does not consolidate test results of several test suites

[boriska70]

I'm running jasmine-reporters with jest - want to create a JUnit XML file that contains all test results for further usage in CI. So I have the configuration like below Package.json - dependencies:

  "react": "^15.1.0",
  "jest-cli": "^12.1.1",
  "jasmine-reporters": "^2.1.1"

Package.json - configuration:

  "jest": {
    "setupTestFrameworkScriptFile": "./setup-jasmine-env.js",
      "collectCoverageOnlyFrom": {
      "./modules/aaa/aaa.js": true,
      "./modules/bbb/bbb.js": true,
      ...
    },
    "unmockedModulePathPatterns": [
      "<rootDir>/node_modules/react/",
      "<rootDir>/node_modules/react-dom/",
      "<rootDir>/node_modules/react-bootstrap/",
      "<rootDir>/node_modules/react-addons-test-utils/",
      "<rootDir>/node_modules/jasmine-reporters"
    ]
  },

setup-jasmine-env:

var jasmineReporters = require('jasmine-reporters');  
jasmine.getEnv().addReporter(
  new jasmineReporters.JUnitXmlReporter({
  consolidateAll: true,
  savePath: './test-results',
  filePrefix: 'test-results-'
  })
);

When I run jest, test run successfully excepting the fact that consolidateAll does not work - at least not as I expected. My expectation was that the single XML file will be created for all my tests suites (each suite is a single test file with single "describe" and with several "it", in my case). In fact the XML report only contains the results of the latest test suite in the run, i.e. the file is recreated for each test suite. I'm a beginner in client-side programming so it may easily be my fault in understanding or configuration; I beg a pardon in advance, if so. Thanks!

[putermancer]

I've never used jest, so I'm not sure if there's something environmental going on there—I may take a stab at setting it up just to try.

To see if there was some obviously broken behavior locally, I tried the following using jasmine-reporters@2.2.0.

1. I split "Some Basic Suite" and "Another Suite" (used in the JUnit example in the examples directory) into individual files which I loaded with <script> tags. Changed that file to use consolidateAll: true. Ran the example with the included phantomjs runner in the bin directory. Result: one file which had specs from both suites.
2. I added a file called reporters.helper.js in the spec/helpers directory, using the config below, and ran jasmine from the command line to have Jasmine's native node.js runner execute all the official specs (JUnit, NUnit, TeamCity). Result: one file with the results from all 3 spec files.

/* global jasmine */
var JUnitXmlReporter = require('../..').JUnitXmlReporter;

jasmine.getEnv().addReporter(new JUnitXmlReporter({
    savePath: '',
    consolidateAll: true,
}));

At this point I suspect something jest-specific, but as I have not used jest before I have not confirmed that yet.

[putermancer]

Can you confirm the behavior still exists for you with v2.2.0 of jasmine-reporters?

[boriska70]

I verified the behavior with v.2.2.0 and simplified project. I have 2 test files with 2 test suites in each. Every test suit has 2 test cases. With consolidateAll set to true, I receive a single junitresults.xml that consolidates all test suites results of a single test file, while the 2nd test file results do not appear in any file. With consolidateAll set to false I get the results of all tests and separate xml result file is created for every test suite (describe function) in all test files. I created a repository that describes the problem so that it should be simple and fast to reproduce it: https://github.com/boriska70/ReactBaseProject . There are test results example for consolidateAll=true use case in test-results folder. One more thing - it can be a jest problem because the text output produced by jest in Node.js command prompt says: 8 tests passed (8 total in 2 test suites, run time 1.577s) while I'd expect to see "...in 4 test suites...".

[reid]

Jest may run your tests using a worker farm of isolated Node processes. This means each instance will write to the same XML file and clobber your other results.

[anas10]

Same issue here with 2.2.0. Did anyone find a solution to this yet?

[putermancer]

From the Jest troubleshooting guide:

Note: the --runInBand cli option makes sure Jest runs test in the same process rather than spawning processes for individual tests. Normally Jest parallelizes test runs across processes but it is hard to debug many processes at the same time.

As suspected above, default Jest behavior appears to be to run tests in parallel using separate node processes. Each process fires up jasmine to run some tests and attaches jasmine-reporters, each process writes to the same file, and your tests get overwritten. You could try running jest with the --runInBand option, which will make it use a single process rather than multiple processes—I do not know for certain that this will solve your problems, because I haven't tested it in Jest and I can conceive the possibility that even in that mode it might run tests serially, but after bootstrapping / configuring Jasmine for each run as it normally would in multi-process mode.

If --runInBand doesn't work, or isn't something you feel comfortable using as a solution, then you probably can't get to a single results output in your Jest environment. As an alternative to consolidateAll: false, one way people get around this issue when using jasmine-reporters as part of parallel Protractor tests is to use something like the environment name or process ID or a random number as part of the filename so the parallel processes don't stomp on each other. This technique, used with consolidateAll: true, results in a smaller number of files than you might get with consolidateAll: false, but still not a single XML file with all test results; you end up with one-file-per-process, however Jest decides to split them up.

One simple way of doing this is to provide a modifyReportFileName function as part of your configuration options. That will call your function before jasmine-reporters tries to write the file, giving you a simple hook for introspecting Jest environment variables that may be useful, generate a random number to avoid collision, etc.

[silentbobbert]

I tried --runInBand and it doesnt appear to stop the results from each suite over-writing each other. I can see the file size growing and shrinking as the tests run. Its always the last test that gets persisted to the file in the end. If you set the consolidateAll: false, you do get a bunch of results, one for each suite - same behaviour reported above.

[bvasilchik]

I'm running into the same issue using protractor and grunt-protractor-runner. If I use consolidateAll: true it's just the last specfile that ran. If I chose consolidateAll: false it makes reports for each describe. I was hoping to get all of them listed together in one .xml file to use with protractor-html-reporter which will generate an html report based off the xml report.

[bloodmonsi]

Same problem here.

[wnadurski]

same here

[sivvoy]

Hi, ditto to all the above and i'm using v4.1.1.

[ViacheslauBushyla]

the same issue in protractor@5.1.0 and jasmine-reporters@2.2.1

[adityagautam]

Still seeing same issue in protractor@5.2.0 and jasmine-reporters@2.2.1. Is there any plans to fix this issue?

[bloodmonsi]

Same here

[egza]

Still seeing same issue in protractor@5.1.1 and jasmine-reporters@2.3.0

[putermancer]

Please consider creating a pull request that addresses the issue encountered when used in these configurations within Jest, Protractor, etc. My time to support this project is greatly reduced from years past. As such, it is unlikely I will find time to investigate and fix this issue myself in the near future, but I'd be very happy to include a fix if someone is able to figure out a good solution that can be shared with the community.