search

mucsi96 / nightwatch-api

[DEPRECATED] Cross-runner API for Nightwatch.js
https://nightwatch-api.netlify.com/
MIT License
84 stars 64 forks source link

closeSession and stopWebDriver failures cause cucumber formatter to fail #596

Open tosborne1215 opened 5 years ago

tosborne1215 commented 5 years ago

There have been previous issues opened about a similar issue, but I think I've seen enough to be able to explain what the problem is.

I've noticed that the failure of closeSession and stopWebDriver can cause the status of a scenario to be undefined. When this happens the cucumber summary formatter then fails. For me that means my pipeline fails and fails to report which makes this a big issue for us.

I do not have steps to reproduce as this is not something that happens due to execution flow. However, when there are connection issues I have seen the issue. Here is an example stack trace that caused the issue.

11:09:55  POST http://hub-cloud.browserstack.com /wd/hub/session/78ad5a74fed85c1538d86ea2e44efe62acf8c288/elements - ECONNRESET
11:09:55 Error: socket hang up
11:09:55    at createHangUpError (_http_client.js:331:15)
11:09:55    at Socket.socketCloseListener (_http_client.js:363:23)
11:09:55   at Socket.socketCloseListener (_http_client.js:363:9)

Here is the cucumber SummaryFormatter exception

11:09:55 TypeError: Cannot read property 'status' of undefined
11:09:55     at SummaryFormatter.isTestCaseFailure (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/formatter/summary_formatter.js:25:106)
11:09:55     at _lodash.default.each.testCase (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/formatter/summary_formatter.js:37:16)
11:09:55     at /data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:4911:15
11:09:55     at baseForOwn (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:2996:24)
11:09:55     at /data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:4880:18
11:09:55     at Function.forEach (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/lodash/lodash.js:9344:14)
11:09:55     at SummaryFormatter.logSummary (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/formatter/summary_formatter.js:36:21)
11:09:55     at EventEmitter.PrettyFormatter.options.eventBroadcaster.on (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber-pretty/index.js:116:17)
11:09:55     at emitOne (events.js:121:20)
11:09:55     at EventEmitter.emit (events.js:211:7)
11:09:55     at Master.onSlaveClose (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/runtime/parallel/master.js:96:29)
11:09:55     at ChildProcess.slave.process.on (/data/jenkins/workspace/mit.nightwatch.acceptance.stubbed.bs.googlePixel3/node_modules/cucumber/lib/runtime/parallel/master.js:83:12)
11:09:55     at emitTwo (events.js:126:13)
11:09:55     at ChildProcess.emit (events.js:214:7)
11:09:55     at maybeClose (internal/child_process.js:925:16)
11:09:55     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

Your Environment

spnraju commented 5 years ago

Hi @tosborne1215 , could you confirm if you still see this issue?

tosborne1215 commented 5 years ago

@spnraju Yes I still experience this. Currently our workaround is hacking the cucumber formatter class to treat undefined statuses as failures. While developing my hack I think I was able to repro the issue or at least cause an issue that was similar.

  1. Run 2 failing scenarios on BrowserStack
  2. After the first failure stop the session through automate.browserstack.com before the 2nd scenario fails.
  3. You will see the error after your last scenario times out.

I think this is just simulating a disconnect from BrowserStack, but I know that this is not the only thing that causes this error. Regardless, cucumber's default reporters expect every scenario to have a status. The fact that I can get an error where the status is undefined is a problem because if that occurs then there is no report.

spnraju commented 5 years ago

@tosborne1215 thank you for the details.