ariya
commented
9 years ago My notes after investigating Sauce Labs integration:
Sauce Labs FOSS plan is really good, it works like a charm with our Karma setup. The only limitation is the maximum number of 5 concurrent VMs per session (and thus 5 browsers in each Karma run). The workaround is easy: just launch Karma with a different configuration in each run. In my forthcoming pull request, there are three different runs which already give a superb browser coverage:
- Evergreen browsers: for latest Edge, Firefox, Chrome, and Chrome Mobile
- Different flavors of Safari, both on OS X and iOS
- Current and past stable versions of Internet Explorer
Note that IE 8 can't run right now due to some issues inside Mocha (among others, because Array#reduce is missing. Probably just needs to be polyfilled).
This Sauce Lab tests are meant to be carried out via drone.io. It is for the same reason that we split our tests to a number of hosted CI (http://esprima.org/test/ci.html): the tests are running in parallel and thus they won't block each other. Typically, all the steps are completed within 10 minutes, give or take.
This multi browser tests are running only on the master branch. We do not want to run it on any other branch, particularly for a pull request, because that means someone might be able to "steal" the access key by reading the environment variable SAUCE_ACCESS_KEY. I believe this is sufficient since our pre-merge checks (see http://esprima.org/test/ci.html) are quite exhaustive, covering IE 11, latest Chrome, and latest Firefox.
Regarding the reliability, it is quite a hit and miss. Sometimes it works for a few runs without any problem, sometimes an explicit build needs to be triggered due to a flaky VM or a broken tunnel or some other random things. Before considering other alternatives (e.g. BrowserStack), let's give this a try and see if the benefits outweigh the minor hassle.
Now that we have a Karma-based test runner for browsers (#1004), we should leverage the "browsers zoo" of Sauce Labs to ensure that Esprima unit tests pass with as many browser as possible.