No error screenshots are taken if mocha tests times out

christian-bromann commented 5 years ago

From @perhenrikandersson on October 1, 2018 7:10

The problem

When running wdio tests with mocha no screenshots are taken if the mocha timeout is reached.

Environment

WebdriverIO version: master (I guess 4.13.2)
Node.js version: v8.11.2
Standalone mode or WDIO Testrunner: Wdio test runner
if wdio testrunner, running synchronous or asynchronous tests: syncronous
Additional wdio packages used (if applicable):
OS: Windows 10

Details

I used the wdio mocha examples to reproduce this.

Step-by-step: 1) Set mochaOpts timeout to 60000 2) Add a browser.waitForVisible('#nonexistingelement',70000); in your test 3) Run tests using wdio wdio.conf.js command Expected: Mocha times out and a screenshot is saved Actual: Mocha times out and no screenshot is taken

If I instead set the mochaOpts timeout to 90000, the test will fail and a screenshot is taken since the mocha timeout is not reached.

Note: I had to run the tests from within the mocha folder using the "wdio wdio.conf.js" command since the command in the README.md didn't work for me (on a windows machine). Yes perhaps I should have written a bug on that too, but that was just not what I was after this time, and I didn't really have time to dig into that problem.

Link to Selenium/WebdriverIO logs

https://gist.github.com/perhenrikandersson/4e511e5c656bdc99406aecd17bae042d

Code To Reproduce Issue [ Good To Have ]

I used the wdio mocha examples: https://github.com/webdriverio/webdriverio/tree/master/examples/wdio/mocha

I modified the wdio.conf.js since I don't have any grid running and I needed errorShots according to: //hostname: '0.0.0.0', //port: 4444, port: '9515', path: '/', services: ['chromedriver'], screenshotPath: './errorShots/',

I also modified mocha.test.js to be able to reproduce the problem according to: it('should have the right title - the fancy generator way', () => { browser.url('http://webdriver.io') // Next row is added by me and is expected to fail browser.waitForVisible('#nonexistingelement',70000); const title = browser.getTitle() assert.equal(title, 'WebdriverIO - WebDriver bindings for Node.js') })

Copied from original issue: webdriverio/webdriverio#2974

christian-bromann commented 5 years ago

From @abjerstedt on December 20, 2018 0:1

Whats the use case for this exactly?

christian-bromann commented 5 years ago

From @perhenrikandersson on December 20, 2018 7:17

Well I always wan't to have a screenshot if my tests fails.

A use case could be that in general I don't expect my test to take longer than 60s. So I set the mocca timeout to 60s.

In my tests I will wait for a splash screen that I won't expect to take longer to load than 45s so I set the timeout for this to 45s. In general it only takes 10s, but I set it to 45s just to be sure. In my tests I then do lets say 5 operations that in general only takes 1s but in some rare occasions they will take 10s each. Since I have many tests I won't set the moccatimeout to e.g. 10 minutes since I would like to have my results faster than that, and i also would like my tests to fail if the total time is more than 60s so I can get a notification that something might be wrong.

So for most of the times my test will take 10s + 1*5s = 15s. But in some cases when I have some slow network or something, my tests will take longer than the mocca timeout of 60s and then I would like to have a screenshot to know how my app looked when the timeout occurred.

webdriverio-boneyard / v4