search

defunctzombie / zuul

[UNMAINTAINED] multi-framework javascript browser testing
958 stars 87 forks source link

Support automated test running in local browsers #248

Open toddself opened 8 years ago

toddself commented 8 years ago

Problem: phantomjs development has slowed significantly and it does not support any ES6 syntax. (Although istanbul also has issues, but I can do --no-coverage). What I'd like to be able to do is use zuul as my test runner locally to run my tests in a local browser (either live or via xvfb, either way is ok with me), like how testling works.

I'm happy to put together a PR for this, but would this be a welcome addition? (Or does this currently work and I just haven't figured it out?)

parshap commented 8 years ago

You can do this using zuul --local.

toddself commented 8 years ago

Wrong idea.

What I want is for zuul to listen to the browser's console, have the browser execute the tests, and then exit returning a non-zero status for failing tests along with console-based output (tap compatible would be great!)

--local requires you to open the browser (or you can use --open if you're lazy) but you have to visually parse the results. The idea here is automated testing using real browsers.

Alternatively phantomjs could kickstart dev again and actually release 2.0....

parshap commented 8 years ago

Ah yes, that'd be great.

I also notice that if you call process.exit() with a non-zero exit code, zuul does not recognize that as a failure.

vvo commented 8 years ago

--local requires you to open the browser (or you can use --open if you're lazy) but you have to visually parse the results. The idea here is automated testing using real browsers.

You can use saucelabs for this. If you really want real browsers local mode (+ exit code etc..) then you should implement selenium connector in zuul and then use selenium.

Otherwise this:

What I'd like to be able to do is use zuul as my test runner locally to run my tests in a local browser (either live or via xvfb, either way is ok with me), like how testling works.

Is exactly what http://karma-runner.github.io/0.13/index.html is

vvo commented 8 years ago

Problem: phantomjs development has slowed significantly and it does not support any ES6 syntax.

I do use --phantom with zuul: https://github.com/defunctzombie/zuul/wiki/Phantom-js But you have to compile to ES5 (BTW you will have to compile to ES5 no matter what you do, because not all browsers are supporting ES6, nor all of it)

toddself commented 8 years ago

phantom no longer runs under os x el capitan, so that's out. the project seems to be on life support (2.0 was never fully released and there is very little activity)

Also most browsers support a decent chunk of es6 at this point; we have a limited number of browsers to support so we're using ES6 syntax that is available in them. Transpiling just for testing is not a scenario I'd like to get into.

Saucelabs costs money and quite a lot of it unless you're working on open source and go through the hassle of opening an account. Even if you're working on open source entirely, I've found saucelabs to be a lot of hassle at times.

I'm like a 1/4 of the way through making a connector to run on a local browser; hopefully will be able to PR shortly.

vvo commented 8 years ago

Transpiling just for testing is not a scenario I'd like to get into.

Indeed, either you always transpile with babel or no transpile at all.

Even if you're working on open source entirely, I've found saucelabs to be a lot of hassle at times.

Yes indeed, mostly selenium is the hassle

vvo commented 8 years ago

I'm like a 1/4 of the way through making a connector to run on a local browser; hopefully will be able to PR shortly.

Sooooo cool! way to go :D

defunctzombie commented 8 years ago

I am happy with removing phantomjs support once this lands. I never really cared for phantomjs support anyway since it was so easy to just open a real browser.

toddself commented 8 years ago

Well this has come full circle. I remember talking with Thorsten to see if he could get phantom re-added as we needed it for local automated testing when we both worked at Conde Nast. :)

Hypercubed commented 8 years ago

I recently switched several projects to zuul (some were using karma others using jsdom). I switched specifically because it is easy to run ES6 code in phantomjs using the babelify transform. Also works well with Travis CI. I hope you will keep phantomjs integration.

Also, sauce labs is ending free accounts.

defunctzombie commented 8 years ago

@Hypercubed hm. Good to know! Also, are all free accounts going away or just for non-open source? If they end all free accounts that would be a bummer.

Hypercubed commented 8 years ago

The FAQ says they are halting Free Accounts: https://support.saucelabs.com/customer/portal/articles/2204230-free-trial-faq But it looks like there is something called Open Sauce.