CLI claims test fails, but dashboard shows it passed

mvasin commented 5 years ago

I followed readme instructions on how to set up, then did npm run parallel.

According to terminal output, firefox test failed:

$ npm run parallel

> nightwatch-browserstack@0.1.0 parallel /Users/mvasin/playground/testing/browserstack/nightwatch-browserstack
> nightwatch -c conf/parallel.conf.js -e chrome,firefox,safari,ie

Started child process for: chrome environment
Started child process for: firefox environment
Started child process for: safari environment
Started child process for: ie environment

  >> chrome environment finished.

  >> firefox environment finished.

  >> ie environment finished.

  >> safari environment finished.

 chrome   [Single Test] Test Suite
 chrome   ========================
 chrome
 chrome   Results for:  Google's Search Functionality
 chrome   ✔ Element <body> was visible after 320 milliseconds.
 chrome   ✔ Testing if the page title equals "BrowserStack - Google Search".
 chrome   OK. 2 assertions passed. (17.815s)
 chrome
 firefox   [Single Test] Test Suite
 firefox   ========================
 firefox
 firefox   Results for:  Google's Search Functionality
 firefox   ✖ Timed out while waiting for element <body> to be visible for 1000 milliseconds.  - expected "visible" but got: not visible
 firefox       at Object.Google's Search Functionality (/Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/tests/single/single_test.js:5:8)
    at Module.call (/Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/nightwatch/lib/runner/module.js:63:34)
    at /Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/nightwatch/lib/runner/testcase.js:97:29
    at _fulfilled (/Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/q/q.js:854:54)
    at /Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/q/q.js:883:30
    at Promise.promise.promiseDispatch (/Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/q/q.js:816:13)
    at /Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/q/q.js:570:49
    at runSingle (/Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/q/q.js:137:13)
    at flush (/Users/mvasin/playground/testing/browserstack/nightwatch-browserstack/node_modules/q/q.js:125:13)
    at process._tickCallback (internal/process/next_tick.js:61:11)
 firefox   FAILED:  1 assertions failed (19.041s)
 firefox
 firefox   ----------------------------------------------------
 firefox   TEST FAILURE: 1 assertions failed, 0 passed (19.088s)
 firefox    ✖ single_test
 firefox   - Google's Search Functionality
 firefox   Timed out while waiting for element <body> to be visible for 1000 milliseconds. - Expected "visible" but got: "not visible"
 firefox
 safari   [Single Test] Test Suite
 safari   ========================
 safari
 safari   Results for:  Google's Search Functionality
 safari   ✔ Element <body> was visible after 323 milliseconds.
 safari   ✔ Testing if the page title equals "BrowserStack - Google Search".
 safari   OK. 2 assertions passed. (21.32s)
 safari
 ie   [Single Test] Test Suite
 ie   ========================
 ie
 ie   Results for:  Google's Search Functionality
 ie   ✔ Element <body> was visible after 1664 milliseconds.
 ie   ✔ Testing if the page title equals "BrowserStack - Google Search".
 ie   OK. 2 assertions passed. (19.167s)
 ie
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! nightwatch-browserstack@0.1.0 parallel: `nightwatch -c conf/parallel.conf.js -e chrome,firefox,safari,ie`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the nightwatch-browserstack@0.1.0 parallel script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/mvasin/.npm/_logs/2018-11-04T18_26_31_612Z-debug.log

but dashboard shows it passed:

gregoryduckworth commented 5 years ago

I believe this is due to the fact that the BrowserStack GUI shows the status of the tests in the "physical" world, in that a session didn't timeout or error.

If it shows completed it means it completed the test but is abstracted away from the result of the test.

I hope this makes sense.

chrisathook commented 4 years ago

that seems incredible useless. I am experiencing something similar where tests are failing but its showing completed in the portal. This pretty much makes this platform useless if you can only observe the actual test results in the console on your development machine.

Varun-garg commented 4 years ago

@mvasin @chrisathook @gregoryduckworth, The reason all tests are green is that the command driver.quit was executed by the user/ framework. The system cannot itself say if the user's test was passed or failed as it does not receive such data during a selenium session.

However, Browserstack provides additional rest API if you would like to mark a session as passed/failed. I think that is what you are looking for. Link: https://www.browserstack.com/automate/rest-api

Someone can patch the framework too to use the above APIs to mark a test as failed.

browserstack / nightwatch-browserstack

CLI claims test fails, but dashboard shows it passed #22