Test running takes increasingly long after a context has finished

[mkj0]

Hi,

I've noticed something strange, but maybe this is the way we're structuring our tests. In order to ensure a clean slate every time, each of our test contexts creates a new browser, then closes it at the end. I've noticed that as we build up the number of tests, the step of closing the browser and carrying on to the next context takes increasing amounts of time as the test run continues. At the very beginning, the step is almost instant. It seems like with every new context that is executed, this step takes a second longer than the one before. I've tried with the LiveHtmlReporter and the ConsolerReporter, both behave the same. It also does the same whether I'm running debug mode in Visual Studio or straight from a command prompt. Any advice or suggestions would be appreciated.

Cheers,

Mike

[lefthandedgoat]

In your before you start chrome and in your after quit() and the more tests you have the longer each quit takes?

[mkj0]

Almost, in our once we start chrome, and close it in the lastly. I have just left a breakpoint on the quit() calls. When I continue running after that breakpoint, that's when the pause happens.

Just in case it happened to be the ChromeDriver, I've just re-run with Firefox and the GeckoDriver. That suffers from the exact same problem.

More edits. :-) The moment the browser does close, the next context fires up very quickly. It's just the closing the browser that's taking a long time.

[lefthandedgoat]

Yes closing a browser does take a long time. If its getting slower to close each time that may be a problem I can address.

I will make a project to try to repro tomorrow and work through it.

[mkj0]

Great, thanks for having a look into it. Out of curiosity I've just stripped everything out of our project down to a single test:

`module Test

open canopy.classic open canopy.runner.classic

let tests () =

context "Tests"

before (fun _ -> start chrome)

after (fun _ -> quit ())

many 30 (fun _ -> url "http://www.google.co.uk" displayed "#lst-ib")`

The only difference here is that the browser is opened and closed in before and after, not once and lastly. It shows broadly the same symptoms, but it's much easier to see the time changes: T1: 4s T2: 8s T3: 10s T4: 12s Tn: 2s longer than the last one

Also a side note, I hadn't noticed previously but this test does make it very obvious, the last test is only marked as successful after the browser has closed, rather than when I would expect, which is before after gets called. If the browser closes quickly, you'ld never notice.

Update: lol, that's 5 times I've tried reformatting the code to look like code. Maybe some VS formatting confusing things. I'm sure you can see what's going on. ;-)

[mkj0]

And the same thing is occurring on my home laptop, so it doesn't appear to be machine specific. I've attached my project, should you want to take a look. Canopy.zip

[mkj0]

Hello, Just an update - a colleague of mine has found that the pause appears to be a problem with the webdriver rather than canopy. Looks like I'm going to have to adapt our tests (thankfully we haven't built a huge amount) to keep the browser open between contexts.

Cheers,

Mike

[lefthandedgoat]

Mike,

Sorry I forgot to look into this. If its an issue with webdriver, make sure yours is up to date with the latest version of chrome. If it is, you may try a few previous versions in case its a recently introduced issue.

I generally just re-use the same browser throughout my tests runs because starting them is slow too.

[mkj0]

Hello again,

Two things now. I'm going to update the tests we have to avoid closing the browser every time, mainly because that probably is a better way for the project to behave, and we also found the problem. The Quit function in canopy.fs (ln 275) isn't cleaning up the list of browsers it has opened. We're not too familiar with F# so we won't be sending you a crude PR, but by removing the current browser from the browsers list, restarting a browser instance was consistently fast every time.

Hope that helps, and thanks for your time.

Mike