search

Victory / django-travis-saucelabs

An Instructive Repo for using Django, Travis and Saucelabs together.
6 stars 0 forks source link

saucelabs tests take longer when running individually #2

Open johanneswilm opened 8 years ago

johanneswilm commented 8 years ago

in your setup you request a new instance for every time, even though you don't switch browsers. This is unfortunately very costly in terms of timing. A tst that used to take 4 minutes now takes about 50 minutes.

Victory commented 8 years ago

That is often a desirable trait to run with a clean app, clears cookies, cache etc...

How would you suggest adding the option to run tests reusing your browser? A global flag, a local flag of some sort or maybe something else?

johanneswilm commented 8 years ago

@Victory: Yes, I can see how this can be desirable in some cases. But just looking at what we currently have [1], where all the saucelabs tests are run in one browser and the total time on travis takes something like 4-6 minutes, and they try to switch to your approach and it goes up to 44 minutes or so. That increase is just so huge that it makes testing kind of unworkable many times.

The reason I wanted to switch away from our current approach is that we are running all tests only in Chrome. Edge and Safari have issues with websockets, but at least I also wanted to include testing in Firefox. Which is why I looked at your example.

It seems like the solution would be to add the option of runnign all tests that use the same browser in the same browser window.

[1] https://github.com/fiduswriter/fiduswriter/blob/3.0/document/tests.py#L36-L50