Testing automation

[jzaefferer]

We want to test this in all supported browsers automatically. I'll look into creating a configuration file for browserstack-runner to see if that works here. Otherwise we can try it with TestSwarm.

[jzaefferer]

@rxaviers can you follow the usage instructions on here? https://github.com/browserstack/browserstack-runner

It should be really easy to get running. Just make sure you've got BROWSERSTACK_USER and BROWSERSTACK_KEY set as env vars.

Would be good to get some feedback from someone who hasn't used the tool before. Please let me know about any issues, no matter how tiny.

[rxaviers]

I assume you have used it before then. Do you have any project with this setup?

I've come up with the below browserstack.json:

{
  "username": "<username>",
  "key": "<key>",
  "test_framework": "qunit",
  "test_path": ["test"],
  "browsers": [
    "chrome_previous",
    "chrome_latest",
    "firefox_previous",
    "firefox_latest",
    "ie_8",
    "ie_9",
    "ie_10",
    "ie_11",
    "ie_latest",
    "opera_12_1",
    "opera_previous",
    "opera_latest",
    "safari_5_1",
    "safari_previous",
    "safari_latest"
  ]
}

Questions: 1)

  "test_framework": "qunit",
  "test_path": ["test"],

Tests should be executed using grunt test. Is the above equivalent?

2)

  "browsers": [...]

Globalize supports http://jquery.com/browser-support/, except IE6 and IE7 right? What are the keywords for mobile? Is config missing any browser entry?

[jzaefferer]

1) Let's get this running first using the browserstack-runner CLI, we can worry about grunt integration later.

2) You don't need both ie_11 and ie_latest. As for mobile browsers, we can add those later, once this is working. Meanwhile I've asked them about the lacks of docs for mobile browser config.

[rxaviers]

Note I am not worried about grunt integration. My concern is whether or not executing browserstack-runner CLI will run the same tests we do via grunt test.

[rxaviers]

You don't need both ie_11 and ie_latest.

Why?

As for mobile browsers, we can add those later, once this is working. Meanwhile I've asked them about the lacks of docs for mobile browser config.

Excellent!

[jzaefferer]

I lost track of our IRC conversation. What was your last result after changing the test_path?

[rxaviers]

http://irc.jquery.org/%23jqueryui-dev/default_%23jqueryui-dev_20140423.log.html#t11:54:32 :)

[jzaefferer]

I just pulled and ran grunt and got a timeout from PhantomJS:

Running "qunit:unit" (qunit) task
Testing http://localhost:9001/unit.html
>> PhantomJS timed out, possibly due to a missing QUnit start() call.
Warning: 1/1 assertions failed (0ms) Use --force to continue.

Seems pointless to run tests through browserstack when they don't even run through PhantomJS.

Once that works, could you retry with browserstack-runner? I suggest to start with just one browser, that will speed things up a lot.

[rxaviers]

It works just fine to me... I have double checked by cloning Globalize into a brand new directory, npm and bower installed dependencies, ran grunt and it worked again. This is my environment https://gist.github.com/rxaviers/11368480.

Could you double check yours please?

PS: What command does browserstack-runner actually run?

[jzaefferer]

I've tracked down the problem within browserstack-runner and sent a PR: https://github.com/browserstack/browserstack-runner/pull/83

Once that is merged and released it should work fine. I had a green run locally without timeouts.

[rxaviers]

:+1:

[rxaviers]

@jzaefferer your PR is merged. Are we good to proceed?

[jzaefferer]

Sure. It was working for me with the patch, so can you give it another try?

[rxaviers]

I've updated my browserstack-runner to 0.1.14. But, I'm getting this error

$ browserstack-runner

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: EACCES, open '/usr/lib/node_modules/browserstack-runner/lib/BrowserStackLocal'

There's no BrowserStackLocal file in my filesystem. Although, I do find the /usr/lib/node_modules/browserstack-runner/lib path.

[rxaviers]

I have pushed my code in here (which is just the config file) https://github.com/jquery/globalize/compare/fix-235-testing-automation?expand=1 in case you need to see it.

[arschmitz]

@rxaviers I ran into this same error working on this for mobile after we talked tonight. Got up to 6 files running and passing in mobile but when I added the file https://github.com/jquery/jquery-mobile/blob/master/tests/unit/page-sections/index.html i started getting this error

[rxaviers]

Thanks @arschmitz for letting me know

[jzaefferer]

Oh well. Can either of you report that against the project itself?

[rxaviers]

Include mobile browsers #278

[rxaviers]

For the record, that's the issue we had https://github.com/browserstack/browserstack-runner/issues/91 .

Now, browserstack-runner works and it submits the tests just fine. Although, the results are inconsistent, ie. tests fail intermittently when ran automatically. But, they pass when ran manually.

[jzaefferer]

370 fixed several issues highlighted by running through browserstack-runner. Adding waitTimeout: 30 to the requirejs config proved to fix the timeout issues, while also speeding up runtime of passing tests (still don't know why tests run faster with a longer timeout).

367 (browserstack.json) still needs some work, should at least add functional tests to the mix.

[rxaviers]

WIP PR #450

[rxaviers]

Close in favor of #450

[rxaviers]

Actually, reopen. Should be fixed by #450.