Set up testswarm.com

[Krinkle]

It's been down for several months now. @jeresig was going to move it to a new server. I'm not sure if that has happened that become swarm.jquery.org or if that is a different initiative.

So is testswarm.com coming back as an open service for approved open-source projects? Or maybe become a redirect to swarm.jquery.org? Will jQuery keep its own swarm or not? etc.

The BrowserStack fill-up for the jquery swarm probably shouldn't be ran on such as an 'open service swarm' (unless it can scale). If such a service would come back it would probably be populated by users all over instead. That is a nice idea if various communities make use of it and members of multiple communities keep the swarm populated. But the catch here is that we've learned in the mean time that opening up the swarm can also cause side effects:

• People joining with fake user agents (either intentionally or unintentionally, e.g. Epiphany matching as Safari)
• Browser clients that are in a bad environment causing false positives (DLL errors, or window not focussed etc.)

Just throwing some random idea out there.

---

Dependencies:

• Domain issues: jquery/infrastructure#38 (internal)
• Design: jquery/testswarm#128

[jzaefferer]

I'd like to update the testswarm.com domain, probably to point to the GitHub repo. I don't think we ever want to setup a public "instance" again, that looks like a rather bad idea by now.

Until I can get access to the domain (along with Google Group, Twitter account), it will stay down...

[jzaefferer]

The domain just got updated and should now point at this repo. Also kicked of transfering it to jQuery infrastructure team, that should make updates easier in the future as well as taking care of extending it.

I'll keep this open until the domain gets updated, doesn't yet load for me.

[Krinkle]

Domain is still unresponsive

• http://testswarm.com/
• http://www.testswarm.com/

[jzaefferer]

Pinging @danheberden about this, he's supposed to have access to the domain by now.

[gnarf]

Looping myself in on this one.

testswarm.com doesn't seem to have any name servers, and is still owned by @jeresig

If we can get @jeresig to point the nameservers set to ns[10-15].dnsmadeeasy.com (copy them from whois jquery.com) - I can set up the DNS records to point wherever you want them to.

Perhaps we should just transfer the entire domain into the account @danheberden controls the main DNS registrars.

[Krinkle]

Domain issues are tracked internally at jquery/infrastructure#38.

Logo, tagline and color scheme are being discussed at #128

[dmr]

What is the status on this issue?

[Krinkle]

Public service

I believe it was decided long ago that a public testing service would not be resumed. The initial beta service built by @jeresig in 2009 was public in two ways: Submitting your own tests, and providing browsers to run tests in. [1]

The public joining of swarms to "donate" your browser resources is still supported in the TestSwarm software. However I believe for any serious use of TestSwarm, it is impractical because there are various issues with how browsers behave under certain conditions. For example, browser extensions often interfere in unreasonable ways. As well as important behaviour differences between a browser being in the background, foreground or minimised.

These and various other issues made jQuery decide back in 2012 that its own instance at http://swarm.jquery.org should have public joining disabled as it constantly caused false negatives in build results. Instead, the swarm is automatically populated as needed by testswarm-browserstack which @clarkbox and I built in 2012.

Allowing anyone to freely submit tests would require a lot of resources that I don't think make sense for the JS Foundation to dedicate in this case, given the small benefit and the large overlap with existing services such as Travis CI and BrowserStack, that already support open-source projects for the same purpose - and would likely be involved either way in most projects that would use this hosted TestSwarm service (e.g. Travis to submit jobs, BrowserStack to launch browsers).

Website

While not as a service, testswarm.com could of course be resurrected as a simple entry point to learn about the TestSwarm software. Similar to https://qunitjs.com and related sites.

Use by jQuery

The software itself works reasonably well and is still in use at http://swarm.jquery.org/ for jQuery core, jQuery UI, jQuery Migrate, and jQuery Color. The browsers are spawned using testswarm-browserstack and the jobs are submitted on each commit from http://jenkins.jquery.com/.

However, most other projects have switched to using only Travis CI, and running the tests directly in a local or remote browser. Typically by using Karma and using local Chrome/Firefox/PhantomJS in the Linux VM, and then running the tests in various remote browsers from SauceLabs or BrowserStack (with a tunnel to access local files) for testing other operating systems, browser versions, and mobile devices.

I very much agree with this model and believe that that is where the future is. There were mainly four features left that I personally liked in TestSwarm and found missing in the typical Travis CI solution when I last looked at this in 2013:

• Secure credentials for cloud browser service.
• Easy overview with which browsers are tested and which failed (if any).
• Easy way to view results of individual browser runs. (HTML report from QUnit)
• Easy re-running of individual builds to deal with race conditions, third-party failures or other intermittent issues and false negatives.

Jenkins + TestSwarm gave us all this. And in 2013 these were either impossible or cumbersome to achieve when using Travis CI directly.

Now, in 2017, these have all been accounted for in various ways, except for a viewer for the individual browser's QUnit test results.

• Travis supports secure credentials (by design not for pull requests, but we don't allow triggering Jenkins jobs on PRs for the same reasons, either).
• Travis has build matrix support (though it doesn't allow splitting results from a single build; which is why most projects still trigger multiple browsers from a single build). Presumably the benefit of a nice overview page is outweighed by the simplicity of having only one build to deal with. However services such as Coveralls make can be used to fetch the results and view them in a different way. Something similar could be done for the test results from Karma/QUnit.
• Travis supports re-running individual builds.

I suppose one could put effort into writing a service like Coveralls that either receives a build artefact, or a static service that lazily interprets the results from the Travis build output and creates a page much like TestSwarm's job and result pages (example).

Until all jQuery projects decide to adopt this model, I'll continue to try and maintain it in spare time where possible. However I don't think it'll develop much further. So setting up a new website doesn't make much sense at this point. If anyone is interested in building the described service or otherwise uses features from TestSwarm not covered by the popular Travis/Karma alternative, I'd love to hear it. Feel free to e-mail me or comment below!

---

Previous write up: https://github.com/jquery/testswarm/issues/205#issuecomment-30887060

[jzaefferer]

Thanks for the writeup and all your work maintaining TestSwarm! I got just one little addition:

Travis supports secure credentials (by design not for pull requests, but we don't allow triggering Jenkins jobs on PRs for the same reasons, either).

For PRs, Travis has the JWT addon: https://docs.travis-ci.com/user/jwt - as far as I know, SauceLabs has implemented it, BrowserStack is working on it.