Run tests in many browsers

[tgvashworth]

fetch-engine should work in the same browsers as the github/fetch polyfill. To make sure this works we should test in many browsers, using browserstack or saucelabs.

Once done, this should be as simple as make test.

[tgvashworth]

Also node, 0.10 -> 5

[tgvashworth]

ava doesn't support running in a browser yet (sindresorhus/ava#24), so we have a bit of a pickle. Work on ava, or switch runner?

[tgvashworth]

I think the options here are:

1. Don't test cross-browser. Yuck.
2. Wait for ava to support browsers. I think this would kill this project
3. Change runner, and figure out how to do testing. Not a huge fan of this, but it might be ok
4. Write a new suite of ("integration") tests that operates on the built javascript but runs in the browser. I kinda like this.

[jackfranklin]

@phuu hope you don't mind me commenting, I was just perusing the project...

I think it would be absolutely reasonable to have a set of almost higher level integration tests that run across browsers. This would mean you can keep the unit tests as is without having to rework them and if you pick carefully your integration tests you should still be able to cover most functionality?

[tgvashworth]

@jackfranklin Happy to see you here!

That's pretty much my thought process. I hate rewriting, and integration tests would give us a nice picture of how the library is to use.

[jackfranklin]

Some Qs/thoughts that may all be stupid!

• Do you want to write the integration tests in TypeScript too?
• Any particular lib of choice? I tend to reach for Jasmine when I need pure browser tests but I guess we really need a good thing that can play ball with browserstack / saucelabs.

[tgvashworth]

• TypeScript: yes please!
• I don't really know. I'm not a fan of Jasmine tbh, and I haven't really looked at testing cross browser with one of those tools. Browserstack have some docs on this. I wonder if we could do a bit of evaluation before picking one?

[tgvashworth]

(And what I mean by that last question is: do you mind having a look?)

[jackfranklin]

I wonder if we go for Tape, compile to TypeScript then browserify or something?! I'll try to do some digging when I get a chance.

[tgvashworth]

That seems reasonable to me. I've enjoyed using tape in the past, and it's quite like ava.

[tgvashworth]

So, we're switching to tape. @jackfranklin did you get a chance to look into browserifying?

[jackfranklin]

@phuu not specifically in this project but I have done Tape + Browserify before, albeit on much smaller project (http://12devsofxmas.co.uk/2015/12/day-2-testing-react-applications/ - see "running in the browser").

I don't have much time going at the moment but happy to share what little I know...

[tgvashworth]

testem looks interesting: testem/testem. They have examples for tape and Browserstack already. I'll have a go at adding it on a branch.

[tgvashworth]

Here's the thing. testem is kinda too much.

The requirements are:

• run the test bundle
• in lots of browsers
• repeatably, from everywhere

tape makes the "run the test bundle" hella easy —it's just console logs that need piping back out to the parent process. Other than that, it needs to:

• start a server to serve the test bundle and a console log consumer
• start a browserstack tunnel
• kick off some browserstack sessions against the local server, through the tunnel
• push the tape output to stdout

While I normally like frameworks & tools that hide this stuff away, I think it's overkill here. Will take a stab at doing this manually.

[tgvashworth]

I had a look at the browserstack-runner but it can't handle tape reporting errors in data structures through its pipes.

Fundamentally, this is what it needs to be as simple as:

$ browserstackify tests/**/*.js
TAP version 13
ok
not ok
...
1..1000

I don't know how you demux TAP output such that something like tap-min can consume it — I guess you have to manually juggle the plan.

Could really do with some help on this one.

[trodrigues]

Hi there!

I can't say I have exactly the same setup but I've recently setup a project using webpack, karma, tape and Sauce Labs. The karma-tap plugin has some issues, like not being able to handle multiple files and is apparently unmaintained but other than that works well.

Karma configs are here and here. The "trick" to fix the problem with multiple files is just requiring them all in one.

Travis config also has some of the sauce labs details. One of the less obvious bits is that when sauce connect is turned on the travis config file, it shouldn't be on the karma runner.

If you're using browserify I assume you just have to make karma use that instead, and it shouldn't be much different.

I also investigated running this with Browserstack when I was trying to set this up but ended up not trying it out because in the meanwhile I figured out we already had an account with Sauce Labs, so I just started using that instead.

[tgvashworth]

@trodrigues Hey! Thanks so much for your response. It seems it's the karma-sauce-launcher that's doing the work to start the browsers there? Unfortunately that's where the problem I have is — the equivalent for browserstack threw as data flowed through it. However, a karma plugin might be different.

While I'd hoped to dodge karma & keep it all ridiculously simple, but I suppose over-simple can get complicated.

[trodrigues]

Yes, exactly. AFAIK testem has a similar thing, so if you've already looked into that you might want to try out first.

Saucelabs only provide a REST API to send your tests and no JS library of their own, so all there seems to be is integrations with Karma, Testem, Grunt and other test/task runners.

[ghost]

Hi @phuu, Maybe I got what you're looking for:

Code: https://github.com/thelambdaparty/fetch-engine Build: https://travis-ci.org/thelambdaparty/fetch-engine

karma + webpack for test running, together with karma-browserstack, seamlessly run tests in BrowserStack. It runs your tests untouched, and has code coverage checking/reporting. (although i would have a second look at tape...)

You can easily add any browsers for testing at karma.browserstack.conf.js, and here is the build output

If you liked it I can send a pull request.

[tgvashworth]

@thelambdaparty Yes!! This is fantastic, thank you so much — please do open a PR.

Quick feedback: would you mind matching the testing/* file's JS-style to the style in the .ts files? Just want to keep things consistent within the project for now.

[tgvashworth]

Fixed by #64.