Update saucelabs tests configuration to run our automated tests on mobile platforms.

[lbod]

@sbrunot can you let me know if all these environments run ok in sauce? The delite tests were hanging in sauce connect using { browserName: "iphone", platform: "OS X 10.8", version: "6.1" } but then I also saw problems with IE browsers using only:

{ browserName: "internet explorer", version: "11", platform: "Windows 8.1" },
{ browserName: "internet explorer", version: "10", platform: "Windows 8" },
{ browserName: "internet explorer", version: "9", platform: "Windows 7" }

and it seemed like when an environment configuration wasn't recognised, sauce connect hanged. I've raised an issue on sauce forums

[sbrunot]

Hi @lbod

I only touched the mobile platform configuration, commenting out the desktop browser environment specs when testing the mobile ones. I was able to launch the android browsers, but not the iOS browsers (hanging, as you experienced).

When I tried to run tests on an unsupported configuration / platform in the past (june 2013), the test failed with the following error reported on SauceLab side: Unsupported OS/browser/version combo: OS: 'Linux', Browser: 'googlechrome', Version: '23.', so I was assuming that the problem with iOS was a network problem (as no test report ever showed up on saucelab site).

Raising the issue on sauce forums is the best thing to do, I think.

[cjolif]

I have a problem with both Android & iOS. For iOS the solution is to go back to what was there before (i.e. not remove "device-orientation": "portrait", "selenium-version": "" information which seems to be important).

Android I have no idea what the right config is but clearly this is not working.

I think @sbrunot you should fix iOS and comment Android until we found the right way to do it.

@lbod I have no problem with the IE environment.

[sbrunot]

Ok, I'll have a look at those configuration and fix this.

[wkeese]

I suggested in email to use Appium rather than WebDriver (for both iOS and android). Appium is apparently the future, whereas AndroidDriver is becoming unsupported.

See https://saucelabs.com/platforms, it tells you how to configure appium vs. webdriver: use the dropdowns on the right and select node.js and appium. But you might have to tweak it's suggested settings some to work with Intern. That's where I get confused.

See also the last comment in https://github.com/theintern/intern/issues/44#issuecomment-34900165.

[sbrunot]

Configurations updated in commit https://github.com/ibm-js/deliteful/commit/91e3a6ef8986b10162c449580091dfcd8b4bca58.

Here are the results of trying to run the tests on various mobile platforms on saucelab:

• android tablet 4.1 => platform and emulator starts, no unit tests running, then timeout on sauce side (Selenium log: NPE at org.openqa.selenium.remote.RemoteWebDriver.getCurrentUrl(RemoteWebDriver.java:291))
• android phone 4.1 => platform and emulator starts, no unit tests running, then timeout on sauce side (Selenium log: NPE at org.openqa.selenium.remote.RemoteWebDriver.getCurrentUrl(RemoteWebDriver.java:291))
• android tablet 4.0 => platform and emulator starts, no unit tests running, then timeout on sauce side (Selenium log: NPE at org.openqa.selenium.remote.RemoteWebDriver.getCurrentUrl(RemoteWebDriver.java:291))
• android phone 4.0: not tested

trying to set selenium-version to "" for android => no platform started using appium is not possible to test in android stock browser.

• iphone 6 => test complete (including functional tests), 11 errors (note: according to the logs, this is using selenium-server)
• ipad 6 => sauce connect error after the unit tests (test error on timeout on sauce side) (note: according to the logs, this is using selenium-server)
• iphone 6.1 => sauce connect error after the unit tests (test error on timeout on sauce side) (note: according to the logs, this is using selenium-server)
• ipad 6.1 => sauce connect error after the unit tests (test error on timeout on sauce side) (note: according to the logs, this is using selenium-server)
• iphone 7 => platform launched on sauce, emulator shows up and then seem to crash while loading www.apple.com in browser (note: according to the logs, this is using appium)
• ipad 7 => platform launched on sauce, emulator shows up and then there is an issue on saucelab side (note: according to the logs, this is using appium)

Trying to directly specify appium for ipad 7 (browserName: "", app: "safari") => platform don't even launch.

[lbod]

@sbrunot thanks, I've linked this issue and your tests from the saucelabs support forum. http://support.saucelabs.com/entries/26491410-ios-timeout-on-10-8-safari-6-1?page=1#post_21906234

I still don't believe it's an ios only problem, I also saw tests that failed (in IE) would successfuly run after restarting sauceconnect

[lbod]

I suggest upping the timeouts on all tests to ~30 secs and for the moment leaving out the android tests (and possibly using only the latest ios, mavericks VM's).

I know sauce have had issues with android emulators and the time it takes to launch but the fact I see no logs at all makes me think it's a sauce problem. I'm unconvinced the ios/android VM's are stable at all in sauce and don't want to waste more time confirming any tests I write, work in them till I know if it's a sauce problem or otherwise.

---

An update from me after wasting a cpl of hours trying to x-browser test the popup functional test in IE. I got consistent failures for waitForCondition failure for: ready with

{ browserName: "chrome", version: "27", platform: [ "Windows 7" ] },
{ browserName: "safari", version: "6", platform: [ "OS X 10.8" ] },

Upping the waitForCondition to 20000 instead of 5000 worked. I then ran

{ browserName: "iphone",
    platform: "OS X 10.8",
    version: "6.1",
    "device-orientation": "portrait",
    "selenium-version": "" } 

using 40000 as the waitForCondition and it ran successfully. I then ran :

{ browserName: "android", platform: "Android" },
{ browserName: "iphone",
    platform: "OS X 10.8",
    version: "6.1",
    "device-orientation": "portrait",
    "selenium-version": "" } 

And the android test didn't run (https://saucelabs.com/tests/9f9ee5e963a048b58fdf87402f930f6b) with 'Test did not see a new command for 30 seconds' but the ios test did (https://saucelabs.com/tests/d26f1f51817843d395fd4f7fc27bfc9d) - no console or sauce logging.

So I ran the android test alone with 40000 as the waitForCondition and that failed (https://saucelabs.com/tests/806fa172a63c420b97fd6220248dffaf) using:

{ browserName: "android", platform: "Linux", version: "4.1", "device-orientation": "portrait"}

You'll see null pointers in the sauce logs :

java.lang.NullPointerException
34  at org.openqa.selenium.remote.RemoteWebDriver.getCurrentUrl(RemoteWebDriver.java:291)

Thinking this was android specific I then ran :

    { browserName: "iphone", platform: "OS X 10.8", version: "6.1"},
    { browserName: "ipad", platform: "OS X 10.8", version: "6.1"},
    { browserName: "iphone", platform: "OS X 10.9", version: "7"},
    { browserName: "ipad", platform: "OS X 10.9", version: "7"},

And the 10.8 platforms failed with The environment you requested was unavailable !!

[wkeese]

Note that (as @lbod pointed out) the way to get iOS working is to remove the capabilities section of the config file (see above commit).

[pruzand]

Note that even with the last configuration from lbod (https://github.com/ibm-js/deliteful/pull/57) we still have a lot of Timeout errors aborting the tests when running against saucelabs, either from office (where we have currently some connections issues) or from home.

[lbod]

@sbrunot @cjolif @wkeese @pruzand @seb-pereira sorry I had to ping you all but didn't know an easier way to group message.

I've seen lots of comments about sauce timeouts, sauce not working, taking a long time etc. The crux of these problems are erring tests, not actual test failures and likely it happens locally as well as remote. What's happening is either the intern/WD waitForCondition is never returning true because the functional client side HTML/JS is erring or your intern functional test itself is failing (and it's always been the former as far as I see) e.g. .waitForCondition("ready", 15000) but the global ready var is never set to true because of an error in your client side test code, something throwing an exception etc

I hope I'm not preaching what you already knew but local & remote webdriver logs will report this. Any sauce logs that report Test did not see a new command for XX seconds or Test exceeded maximum duration.... means the test code itself errored, not a test failure and nearly always likely to be the .waitForCondition

[sbrunot]

@lbod Yes, I think we're all fully aware of that. The problem is that we don't have any clues of why a page is not ready after the timeout when it's not. As you say, it can be because of an error thrown while creating the page content. It can also be that the connection is so slow that the page and its dependencies are never loaded before the timeout occurs. We have been pretty successfull at solving tests issue relying on this last hypothese so far, and all our tests are running successfully at the moment on saucelabs desktop browsers, except, apparently, when the tests are launched from the travis CI platform... Any idea of how we could enhance our tests to retrieve the browser console output when the test fails ?

[wkeese]

Any idea of how we could enhance our tests to retrieve the browser console output when the test fails ?

I think this works already, but the problem is that there's no console output.

If you go to https://saucelabs.com/u/ibmjs and then click one of the failed tests (for example this one), IIUC any console output will be in the left pane listing all the HTTP requests.

[sbrunot]

Since commit https://github.com/ibm-js/deliteful/commit/5b34ce337cd965900e748f66ad8dceed15c1376b, deliteful tests are now running on SauceLab iOS 7 platforms.

[lbod]

Sorry this is long but I wanted to show how I debug these issues.

If you go to https://saucelabs.com/u/ibmjs and then click one of the failed tests (for example this one), IIUC any console output will be in the left pane listing all the HTTP requests.

Bill's right you can view some console logging in Sauce itself but it's probably not verbose enough for these types of issues.

What I do to debug errors like this is first look at the Travis output e.g. https://travis-ci.org/ibm-js/deliteful/builds/24799079#L1108 on line 1108 it says

at Test.registerSuite.read only ltr [as test] (/home/travis/build/ibm-js/deliteful/tests/functional/StarRating.js:168:5)

Which means it's failing at https://github.com/ibm-js/deliteful/blob/master/tests/functional/StarRating.js#L168

If you don't know already, you can actually interact with the running Saucelabs test. I can't remember if you need to be logged in to Sauce to interact or if you can do this via https://saucelabs.com/u/ibmjs but when a test like this is stalled, click the live video to interact. Press F12 or whatever to view the console and see what JS errors are output. You can also reload the test by reloading the URL in the browser, so you can set an exception break-point in the debugger tools and debug where it fails.

I still find this statement strange

our tests are running successfully at the moment on saucelabs desktop browsers, except, apparently, when the tests are launched from the travis CI platform

I can't see there being that much latency between the Travis VM's and Sauce that it'd cause tests to time out. If grunt test:remote works from your local machines, I can't see this being any different to the same command run from Travis.

You may also want to change the .travis.yml script to run grunt jshint test:remote:console just to get better test reporting output in Travis but it'll only help tell you which test it's erroring in, which you should already know anyway. It's a bit too verbose which Bill/Christophe didn't want so it was removed.

---

Anyway, I ran a bower update on deliteful then ran grunt jshint test:remote from my own machine, commenting out all tests apart from StarRating, I should be running the most upto date code (I only quickly tried this so I could be wrong). However it looks like this is the same issue with bower where tests files are excluded (in delite) and deliteful is sharing test files with delite. It now makes sense that you're seeing errors run from Travis compared to running the Sauce remote test from your own machine i.e. because the Travis script runs bower install and because tests are excluded from delite, it doesn't copy those files.

I see a 404 for delite/tests/boilerplate.js. I raised an issue about this before (which I can't find) and Adrian fixed https://github.com/ibm-js/deliteful/commit/833a07752ee5457afa03e34643cb9c157f0fa740 So I think you'll need to duplicate boilerplate.js in deliteful

e.g. see

I'm just trying to highlight there are ways to investigate these issues which are fairly easy to do & I apologise but I don't like seeing issues reported miscellaneously due to supposed "slow connections" Hope that helps

[pruzand]

@lbod thanks a lot for your time on investigating these tests issues. They are very helpful investigations and informations (I personally did not realize the level of interaction we can have with the SauceLabs VMs).

Regarding the boilerplate.js sharing, @sbrunot please could you check it indeed solves the Travis issue (with your own deliteful clone for ex.) (sounds like it is indeed the reason) and if so, report your findings to https://github.com/ibm-js/deliteful/issues/113 ?

[sbrunot]

@pruzand @lbod Thanks guys for your investigations on that. I can surely check that using a dedicated boilerplate.js file for deliteful solves the issue. Can somebody direct me to the procedure to follow to run a travis build and how it is configured (deliteful build documentation) ?

Also, I think they are too many different issues discussed in this ticket: this ticket is about running our tests on sauce labs mobile configuration. I think we should also have a ticket dedicated to the travis CI build itself: as @Ibod pointed out, running tests during the travis CI build is not equivalent to running tests using grunt:build on our laptop, and we have been talking about different things all along ("it works on my machine" syndrome, because we were doing different things).

[cjolif]

isn't the solution to drop using boilerplate instead of duplicating it to avoid a 404? I thought the purpose of it was to load either RequireJS or Dojo loaded but we decided to go with RequireJS for testing so it should not be necessary anymore.

cc: @wkeese

[sbrunot]

@cjolif If I'm correct, boilerplate.js does a little more than that: it allow RTL bidi testing by adding dir=rtl to the test page URL, and I think it also allows for some a11y testing related to high contrast mode.

[wkeese]

@sbrunot is right that boilerplate has many purposes. I don't think it's doing anything too critical though. I have mixed feelings about whether we should continue to use it or not.

The a11y thing is probably no longer relevant, because our widgets shouldn't be branching based on high-contrast mode. If they just use font-icons then they will show up fine regardless of the mode.

The RTL flag is nice but not strictly necessary. You can always put two widgets in your test file (one with dir=ltr and one with dir=rtl). I don't know, maybe that would be too much bloat.

Also, previously boilerplate.js dealt with picking the theme based on a URL param, but now that's done by delite/theme.js.

About loaders, it's still nice to be able to test with different loaders, even though we are concentrating on RequireJS. Or users will be using different loaders and eventually we will want to fix cross-compatibility issues in our code.

[pruzand]

Personaly I would be in favor to keep boilerplate for both RTL and loaders testing purposes. The RTL flag is in fact both my main motivation to keep it... but I wonder how the wd instructions based on mouse-coordinates work with rtl: for ex, moveTo() that takes offsets related to top-left corner of an element. Because if it does not, that means the tests need to take into account rtl, and that would defeat the benefit of the boilerplate rtl capability.