Intern 2.0.1 test failing on safari iOS (SauceLabs)

[sbrunot]

I've tried to run a simple functional test that only display the google.com homepage:

define([
    "intern!object"
], function (registerSuite) {

    registerSuite({
        name: "Sample test suite",
        "sample test": function () {
            return this.remote
                .get("http://www.google.com");
        },
    });
});

It is then run using the SauceLabs tunnel, requesting the following capabilities (intern environments configuration):

environments: [
        { browserName: "", platform: "OS X 10.9", version: "7.1", deviceName: "iPhone",
            app: "safari", device: "iPhone Simulator", name: "sample test" }
]

Unfortunately, this simple test fails with the following error:

Warning: Test main - Sample test suite - sample test FAILED on Any browser on MAC:
UnknownCommand: [POST http://sbrunot:d9b67b40-76ed-4237-8042-f5d582e2c29d@localhost:4444/wd/hub/session/a889e01b9e104e79894e703307c34cf8/execute / {"script":"return (function getCoverageData() {\n\t\t/*global __internCoverage:false */\n\t\treturn typeof __internCoverage !== 'undefined' && JSON.stringify(__internCoverage);\n\t}).apply(this, arguments);","args":[]}] The requested resource could not be found, or a request was received using an HTTP method that is not supported by the mapped resource.
  at Server._post  <node_modules/intern/node_modules/leadfoot/Server.js:62:9>
  at ProxiedSession._post  <node_modules/intern/node_modules/leadfoot/Session.js:57:30>
  at ProxiedSession.Session.execute  <node_modules/intern/node_modules/leadfoot/Session.js:580:15>
  at ProxiedSession.lang.mixin.get  <node_modules/intern/lib/ProxiedSession.js:77:17>
  at CompatCommand.<anonymous>  <node_modules/intern/node_modules/leadfoot/Command.js:617:19>
  at <node_modules/intern/node_modules/leadfoot/node_modules/dojo/Promise.js:256:37>
  at runCallbacks  <node_modules/intern/node_modules/leadfoot/node_modules/dojo/Promise.js:13:18>
  at <node_modules/intern/node_modules/leadfoot/node_modules/dojo/Promise.js:204:17>
  at Object.run [as _onImmediate]  <node_modules/intern/node_modules/leadfoot/node_modules/dojo/Promise.js:143:29>
  at processImmediate [as _immediateCallback]  <timers.js:330:15>
  at new CompatCommand  <node_modules/intern/runner.js:195:14>
  at <CompatCommand.target.(anonymous function) [as get] (/Users/sbrunot/Github/deliteful/node_modules/intern/node_modules/leadfoot/Command.js:592:11)>
  at Test.registerSuite.sample test [as test]  <tests/functional/FunctionalTest.js:9:6>
  at Test.run  </Users/sbrunot/Github/deliteful/node_modules/intern/lib/Test.js:154:18>
  at <node_modules/intern/lib/Suite.js:212:13>
  at signalListener  <node_modules/intern/node_modules/dojo/Deferred.js:37:21>
  at Promise.then.promise.then  <node_modules/intern/node_modules/dojo/Deferred.js:258:5>
  at <node_modules/intern/lib/Suite.js:211:46>
  at process._tickCallback  <node.js:415:13>
Error: Test did not see a new command for 60 seconds. Timing out.
  at <node_modules/intern/node_modules/digdug/SauceLabsTunnel.js:321:12>
  at <node_modules/intern/node_modules/digdug/node_modules/dojo/Promise.js:256:37>
  at runCallbacks  <node_modules/intern/node_modules/digdug/node_modules/dojo/Promise.js:13:18>
  at <node_modules/intern/node_modules/digdug/node_modules/dojo/Promise.js:204:17>
  at Object.run [as _onImmediate]  <node_modules/intern/node_modules/digdug/node_modules/dojo/Promise.js:143:29>
  at processImmediate [as _immediateCallback]  <timers.js:330:15>
  at new CompatCommand  <node_modules/intern/runner.js:195:14>
  at CompatCommand.Command.then  <node_modules/intern/node_modules/leadfoot/Command.js:525:10>
  at CompatCommand.Command  <node_modules/intern/node_modules/leadfoot/Command.js:549:15>
  at Suite.teardown  <node_modules/intern/runner.js:237:36>
  at callOnSuite  <node_modules/intern/lib/Suite.js:131:42>
  at call  <node_modules/intern/lib/Suite.js:182:13>
  at finishRun  <node_modules/intern/lib/Suite.js:240:6>
  at <node_modules/intern/lib/Suite.js:219:7>
  at process._tickCallback  <node.js:415:13>

[csnover]

Well, we’re not able to support a WebDriver server that doesn’t support the standard execute function for hopefully obvious reasons, so I am not sure we’ll be able to do anything about this. Sauce Labs never had a really functioning iOS testing service; the only driver approaching usability with iOS is ios-driver, and Sauce don’t use it. IPhoneDriver is deprecated, and Appium just isn’t up to the task (it is missing way too many APIs).

[sbrunot]

For the record, it works fine using intern 1.6.

[lbod]

@csnover it sounds like you're saying iOS tests are currently not supported (well), just so we can focus our testing efforts better. I do see the same in delite where tests will fail in intern 2.0.1 but run in 1.7.

It appears both Android and iOS emulators/simulators aren't stable and not worth testing against? SauceLabs have said they plan to have real device testing and selendroid (for chrome testing) by the end of the year so maybe that's the point when mobile web testing will be supported

[sbrunot]

For the record, I've performed some further experiments and noticed that the test runs fine when using NullTunnel and a local instance of appium. So this is really a SauceLab related issue, not an appium issue.

[sbrunot]

I've narrowed down the issue to the features and defects discovery process of leadfoot/Server.js.

If I comment out the two following pieces of code in Server.js:

// At least ios-driver 0.6.6-SNAPSHOT April 2014 does not support execute_async
testedCapabilities.supportsExecuteAsync = session.executeAsync('arguments[0](true);').catch(unsupported);

and

// At least ios-driver 0.6.6-SNAPSHOT April 2014 will never complete a refresh call
testedCapabilities.brokenRefresh = function () {
    return session.get('about:blank?1').then(function () {
        var dfd = new Promise.Deferred();
        function cleanup() {
            clearTimeout(timer);
            if (!refresh.isFulfilled()) {
                refresh.cancel();
            }
        }

        var refresh = session.refresh().then(function () {
            cleanup();
            dfd.resolve(false);
        }, function () {
            cleanup();
            dfd.resolve(true);
        });

        var timer = setTimeout(function () {
            cleanup();
        }, 2000);

        return dfd.promise;
    }).catch(broken);
};

the test runs fine, without any issue.

[csnover]

To confirm, this is a problem with Appium? If so we can merge this into #223.

Otherwise, there’s no plan to support the deprecated IPhoneDriver that got superseded by ios-driver and Appium, since it’s too broken in other areas to be treated as a useful test environment.

With regards to the issue only happening in Sauce Labs cloud specifically, that makes it difficult for us to fix since it’s not a compatibility issue with the driver itself, but an issue with the cloud provider. We have lots of these issues with Sauce, not exclusive to their mobile testing stuff, which is one of the reasons why support for other providers was added in 2.0. I am not sure what our long-term plan needs to be to try to handle issues like these, since their team consistently fails to address known issues or communicate with us in any way.

[admc]

FWIW, we are aware of the frustrations some folks are having with Intern and Sauce, and have a roadmap project in the coming weeks to dig in and contribute fixes to as many as we can. To be clear, our cloud supports IOS simulators and Android emulators, each of which has many successful users. We will also be making real devices available in the coming months.

[jlipps]

Which appiumVersion are these tests sending in the desired caps? The minimal version expected to work on Sauce is appium 1.2.1 (this is the first version of appium that allows executeAsync to work properly when appium is on a different machine than the client).

[csnover]

@admc Great that you are aware and looking into it! Could you guys maybe respond now to the ticket that has been open since March about how your monocle proxy is broken (10498)? We asked for a status update on July 28 and have received no response.

[sbrunot]

@jlipps thanks for your feedback. No specific capability is set to request a particular appium version. I'll update the test to specifically request appium 1.2.1 and see if it fixes the issue.

[jlipps]

OK, sounds good @sbrunot. Better yet, request 1.2.2 (just released that new version on Sauce)

[sbrunot]

@jlipps thanks to your help (and this blog post https://support.saucelabs.com/entries/50675504-iOS-Desired-Capabilities), I've updated the capabilities as following to run the test using appium 1.2.2:

{ platformName: 'iOS', platformVersion: '7.1', browserName: 'safari', deviceName: 'iPhone Simulator', "appium-version": "1.2.2" }

Unfortunately, it also failed, but for a different reason: Intern does not report any failure (it doesn't report anything), but I can see on the SauceLab side that the test failed (the Saucelabs test execution result is here (I guess it is a public one): https://saucelabs.com/tests/dacc96e8173a42ae88a8aba5db82cd68).

I haven't had time to debug Intern to understand at which stage it is failing this time.

[jlipps]

not sure what's going on there. looking at the logs, you call a refresh() command after navigating to "about:blank?1", then we don't hear from you for 60s so the server ends the session. is there a reason your client would be stalling at that point?

[csnover]

Yes, a bug at this line: https://github.com/theintern/leadfoot/blob/master/Server.js#L721 (the isFulfilled call is invalid, it should just always call cancel now)

[sbrunot]

I confirm that with @csnover 's fix, the test runs fine.

[jlipps]

great!

[sbrunot]

Thanks a lot.