Cucumber reports fail to create JsonOutput

samhatoum commented 6 years ago

Issue by asanchezgiraldo Sunday Mar 19, 2017 at 21:44 GMT Originally opened as https://github.com/xolvio/chimp/issues/596

Expected behaviour

When chimp is run with the command --jsonOutput=reports/regression.json a json file regression.json should be created with all of the scenarios executed

Actual behaviour

Chimp fails due to values of JSON being null

Version & tools:

Chimp command line used: ./node_modules/.bin/chimp chimp.js --browser=phantomjs --format=rerun:@rerun.txt --jsonOutput=reports/regression.json --tags=@now --tags=@must

Chimp config file:

module.exports = {
//Generic config
tags: '@'+tag_id[0],
screenshotsOnError: true,
captureAllStepScreenshots: false,
saveScreenshotsToReport: false,
screenshotsPath: 'screenshots',
saveScreenshotsToDisk: true,
webdriverio: {
    desiredCapabilities: {}
}
};

Chimp: 0.47.2
Node.js: 6.1.0
Java: 1.8.0_102
Operation system: Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64

Console / Log Output

node_modules/chimp/dist/lib/cucumberjs/cucumber.js:101
          if (_this.options.jsonOutput && JSON.parse(jsonResults).length) {
                                                                 ^

TypeError: Cannot read property 'length' of null
    at ChildProcess.<anonymous> (/Users/alejandro/now-site/automation/node_modules/chimp/dist/lib/cucumberjs/cucumber.js:101:66)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:850:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:215:5)

Join our Slack xolv.io/community #chimp channel, where you can find help and help others.

samhatoum commented 6 years ago

Comment by tiagolr Tuesday Mar 21, 2017 at 18:03 GMT

+1 also featured here: https://github.com/xolvio/chimp/issues/575

By disabling screenshots entirely I'm able to get the json output without flaws, with screenshots it really depends on flags such as --watch, embedScreenshots and test sizes, would be great to have embed screenshots without failures in the future, also if reading this, currently screenshots are taken for skipped tests i think, those could be removed too.

samhatoum commented 6 years ago

Comment by elenadimitrova Monday Mar 27, 2017 at 17:07 GMT

Running chimp with env variable for CI , which sets the screenshotsOnError property https://github.com/xolvio/chimp/blob/master/src/bin/default.js#L28 still gives me an error when running with --jsonOutput option:

>> /home/ubuntu/appRoot/node_modules/chimp/node_modules/cucumber/lib/cucumber/runtime/event_broadcaster.js:30
>>             throw error;
>>             ^
>> 
>> Error: node_modules/chimp/node_modules/cucumber/lib/cucumber/support_code/library.js:17 [chimp] server.execute timeout after 60000ms
>>     at null._onTimeout (/home/ubuntu/appRoot/node_modules/chimp/dist/lib/chimp-helper.js:186:19)
>>     at Timer.listOnTimeout (timers.js:92:15)
>> /home/ubuntu/appRoot/node_modules/chimp/dist/lib/cucumberjs/cucumber.js:101
>> if (_this.options.jsonOutput && JSON.parse(jsonResults).length) {
>> ^
>> 
>> TypeError: Cannot read property 'length' of null
>>     at ChildProcess.<anonymous> (/home/ubuntu/appRoot/node_modules/chimp/dist/lib/cucumberjs/cucumber.js:101:66)
>>     at emitTwo (events.js:87:13)
>>     at ChildProcess.emit (events.js:172:7)
>>     at maybeClose (internal/child_process.js:829:16)
>> at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)

samhatoum commented 6 years ago

Comment by cadavre Wednesday Apr 05, 2017 at 12:58 GMT

Same error here. I have no report due to this error.

After short debugging, looks like this: cucumber.js:89 this.cucumberChild.on('message', function (res) { is never called, so no message "message" is being received in Chimp.

Update: happens only in watch mode.

Update 2: report is generated without watch mode + screenshots are created if screenshotsOnError: true,. Now in report.json I can see error messages. Still Failures: at the end of testing is empty.

samhatoum commented 6 years ago

Comment by dirwin517 Monday Apr 10, 2017 at 18:15 GMT

Not Ideal, but as a temp fix you can, do your own report file, and comment out the jsonOutput part of chimp config

Just make a File in your step_definitions folder with the following content

Code:

module.exports = function JsonOutputHook() {
 var Cucumber = require('cucumber');
 var JsonFormatter = Cucumber.Listener.JsonFormatter();
 var fs = require('fs');

 JsonFormatter.log = function (json) {

   try {
     fs.writeFileSync(process.cwd() + 'report.json', json);
   }
   catch(err){
     console.error('error:', err);
   }
 };
 this.registerListener(JsonFormatter);
};

samhatoum commented 6 years ago

Comment by tiagolr Wednesday Apr 19, 2017 at 22:43 GMT

Great hack, I can live without the screenshots which cause the json output to fail, but good to see that.

samhatoum commented 6 years ago

Comment by tiagolr Friday Apr 21, 2017 at 00:29 GMT

great work, seems fixed on 0.48

samhatoum commented 6 years ago

Comment by tiagolr Friday Apr 28, 2017 at 14:44 GMT

Correction, still happening on CI, hard to reproduce locally tho, happens at the beginning of a feature rather than at the end of tests.

build   28-Apr-2017 06:49:43    [chimp] Running...[39m
error   28-Apr-2017 06:50:52    
error   28-Apr-2017 06:50:52    /data/build-dir/BNR-E2E-TES/e2e-tests/node_modules/chimp/node_modules/cucumber/lib/cucumber/runtime/event_broadcaster.js:30
error   28-Apr-2017 06:50:52                throw error;
error   28-Apr-2017 06:50:52                ^
error   28-Apr-2017 06:50:52    node_modules/chimp/dist/lib/cucumberjs/hooks.js:20 function timed out after 60000 milliseconds
error   28-Apr-2017 06:50:52    /data/build-dir/BNR-E2E-TES/e2e-tests/node_modules/chimp/dist/lib/cucumberjs/cucumber.js:115
error   28-Apr-2017 06:50:52              var noTestsFound = JSON.parse(jsonResults).length === 0;
error   28-Apr-2017 06:50:52                                                        ^
error   28-Apr-2017 06:50:52    
error   28-Apr-2017 06:50:52    TypeError: Cannot read property 'length' of null
error   28-Apr-2017 06:50:52        at ChildProcess.<anonymous> (/data/build-dir/BNR-E2E-TES/e2e-tests/node_modules/chimp/dist/lib/cucumberjs/cucumber.js:115:53)
error   28-Apr-2017 06:50:52        at emitTwo (events.js:87:13)
error   28-Apr-2017 06:50:52        at ChildProcess.emit (events.js:172:7)
error   28-Apr-2017 06:50:52        at maybeClose (internal/child_process.js:862:16)
error   28-Apr-2017 06:50:52        at Process.ChildProcess._handle.onexit (internal/child_process.js:222:5)
build   28-Apr-2017 06:50:53    [32m

Using @dirwin517 hack for now.

Edit - the error is related to: #616

TheBrainFamily / chimpy