Sauce and fixes

[cscott]

I think I've fixed all your outstanding notes now. Hopefully the travis build will pass! (There seems to be an intermittent error -- maybe a race -- with one of the set tests. I'm looking into it.)

[cscott]

OK, seems like 939ff60 did the trick for the transient failures. And all the sauce tests are passing, whoo! All the way back to IE9.

[ljharb]

Awesome, I've been trying to figure out that transient failure for awhile! :-D

How is https://saucelabs.com/u/es6-shim configured? Can access be shared across multiple accounts? If I wanted to play with adding more browser versions, how would I do that?

If/when the tests are migrated away from mocha to tape, will they continue to work on sauce?

[cscott]

I think I can create a subaccount for you on saucelabs. I haven't played with that yet. Otherwise I can just email you the login info.

Saucelabs just runs the test/index.html file. You can use any test system, although saucelabs has some special support for mocha (and jasmine and a few others). But you can just set a global variable on the window to the results of the test if needed (I do that in cscott/togetherjs).

To try new versions, the easiest way at the moment is to edit the list in Gruntfile.js and then push to a branch on github, and let travis take care of triggering sauce. git push -f origin ljharb-test for example. Once we sort out the subaccount/login details, you can set SAUCE_USERNAME and SAUCE_ACCESS_KEY in your environment and then run npm run sauce or grunt sauce from the command-line. I actually prefer using travis to trigger things because that way they get tagged with a build number in the sauce interface, which helps keep the different test runs separate.

[cscott]

I sent you an invite link to saucelabs using a subaccount. I suggest making your username something with 'es6-shim' in it (I used cscott-es6-shim) so that you can keep the subaccounts for other saucelabs-using projects separate (for example, I also have cscott-togetherjs). You can then later link this to your github account ("edit settings" in the left hand bar, then scroll down to "github integeration") if you'd rather not remember a separate login. I'm not entirely sure how github integration works if you have different per-project accounts, though.

And then create a local file named something like sauce-env.sh with:

export SAUCE_HAS_TUNNEL=true
export SAUCE_USERNAME=ljharb-es6-shim
export SAUCE_ACCESS_KEY=<something>
export TRAVIS_JOB_NUMBER=ljharb

where the SAUCE_ACCESS_KEY is the long hex string shown in the bottom of the left-hand menu bar once you log in to saucelabs.com. To start sauce tasks locally (these connect to your local machine via a tunnel to read the index.html files & etc), start Sauce Connect with the -i ljharb option (this matches the string set in TRAVIS_JOB_NUMBER above), then source sauce-env.sh and npm run sauce. npm run sauce will start a local web server on port 9999 which saucelabs will connect to via sauce connect.

Using identified tunnels is important because otherwise sauce connects to whatever currently-running tunnel it feels like, which can lead to surprising results. Although this shouldn't happen, I think even currently-running travis-triggered tests can end up connecting to your local machine if you're running a tunnel without the -i option.

[ljharb]

Thanks, I logged in to my subaccount, but it doesn't appear to give me access to the previous test runs - it looks like it's just my own sandbox, using the parent account's minutes. We can play with that over time; ideally just like in Travis, we can all view the same set of sauce results.

Overall this looks good - it's unfortunate how much config is required; testling is much simpler - but we definitely need automated browser testing, and this seems to be working well :-) Thanks!

[ljharb]

@cscott It looks like this integration has stopped working: https://travis-ci.org/paulmillr/es6-shim/jobs/58485773 I'm not sure why. Can you look into it?