Sauce Labs Automated Testing

[bnjmnt4n]

Travis CI has already been integrated into JSON3, so I think we can start to implement automated browser testing with Sauce Labs. Lo-Dash has been doing this for some time already, with great results. The only problem with implementing this is that Spec, the test suite we use, is not natively supported, meaning we have to use wd (Selenium/WebDriver library) to access the test results ourselves.

[bnjmnt4n]

I'll work on this on the saucelabs branch with my saucelabs info for now

[jdalton]

Why hide the sauce username? I assume you'll have badges or link to the results page, which exposes the sauce username anyways, it's a public project after all.

[bnjmnt4n]

Yeah, I was using my username first. Eventually I'll change that.

[jdalton]

Cool.

[ghost]

Let's think about migrating away from Spec if it makes Sauce integration easier.

[jdalton]

Sauce supports custom unit test libs :D :goat:

[bnjmnt4n]

Yeah, it does, was a bit busy. There’s no need to switch from Spec. Will try to finish soon.

[ghost]

Cool. No rush at all; just curious if switching to something like QUnit would make things easier. Thanks, @jdalton and @d10! :smiley_cat:

[bnjmnt4n]

I made a support request on Sauce Labs, since the tests worked fine, yet they were failing on Sauce Labs. One of the awesome people at Sauce Labs, @saucyallison, responded:

I took a look at your test, and then did some testing of my own. I don't see anything wrong with your test, but I think I've found the reason your test results aren't displaying properly. I took the contents of your window.global_test_results and put them on a sandbox html page so I could use it with the JS unit test API. My tests were returning "Finished" instead of the proper results like your tests were. I noticed that the window.global_test_results was a fairly large object, so I tried removing test info from it. Eventually, with enough info removed, the unit test results display as expected, with Pass/Fail status. I think the underlying problem here is that the window.global_test_results are stored in the "custom data" field in our database (and on the Metadata tab in the test details page), which has a maximum size. Once this limit is reached, the custom data field gets set to null instead. The size limit is supposed to be 64 KB, but it looks like the size of your input is closer to 42KB, so I believe there's something wrong with our logic. We're looking into increasing the size of this field in the near future! In the meantime, my advice would be to cut your tests down into smaller chunks. Here is an example of the largest global_test_results object I was able to get working: https://saucelabs.com/tests/ffef55684c3748c683db30e8d9bf3e56 For this one, I added one additional unit test, and the results stopped displaying correctly: https://saucelabs.com/tests/b9c9c9461db7409b8bb789dfa4bc45ab Hope this helps! Sorry for the confusion here, I know this kind of problem isn't very self-explanatory. Let me know if you have further questions, thank you!

For now, I'll just not add passing tests to the global_test_results variable, until @saucelabs fixes this.

[bnjmnt4n]

@kitcambridge This now works. Feel free to rebase/merge this. (Test fails are not related, they are due to the Rhino download.)

[ghost]

This is awesome. :star2: Thanks so much, @d10! I'll merge tonight.

[ghost]

"Three weeks" is an interesting definition of "tonight." :crying_cat_face: Sorry for letting this slide; I've been really busy. Merging now.

[jdalton]

Booo GoLang Boo! : D

[ghost]

@jdalton Go was four weeks ago...it's Twisted Python now. :stuck_out_tongue_winking_eye:

[ghost]

Merged as 2043202f4f1f73f6a085c4852dbafb2c43e847d0. Thanks again for the great work, @d10. Let's see if #78 fixes the Date-related failures...

[bnjmnt4n]

Great banter, m8! ;)