Tunnel to Saucelabs closes quickly and does not run tests

[jansepar]

I recently upgraded to grunt-saucelabs 8.1.1 because the previous version (4.0.4) I was running seems to no longer be compatible with the Saucelabs endpoint. Unfortunately, I can no longer seem to get my tests up and running. The output for me looks like the following:

Running "saucelabs-mocha:all" (saucelabs-mocha) task
=> Starting Tunnel to Sauce Labs
=> Stopping Tunnel to Sauce Labs

I tried adding onTestComplete to my config, but that didn't seem to change anything. Help would be much appreciated, as this is probably my favorite/most useful grunt plugin :)

[Jonahss]

Hiya,

I'm just guessing that you haven't added the self-reporting javascript to your test page. Check out examples on the README. Basically, the plugin now uses the Sauce Unit Test API, which requires the js on your page to expose the test results to a global variable, for our servers to pick up on.

Feel free to email me or post your sauce username here, and I can try to figure out what else could be going wrong.

[jansepar]

Ah, so I realized one issue I had was that I didn't update my Saucelabs key after getting it reset :)

Now the tests seem to run, but one interesting thing is that the tests tell me they all pass (which is correct after looking at the tests!), but in Saucelabs, I get the following error for every test:

Test exceeded maximum duration after 180 seconds

Any idea what is going on there? Could it be because I'm running 16 concurrent?

[Jonahss]

Do the tests report as passing on the command-line for grunt-saucelabs? I still think you aren't including the self-reporting javascript on your test pages.

[jansepar]

I added the following snippet in my index.html file in the head:

<script>
 +      onload = function(){
 +        //mocha.checkLeaks();
 +        //mocha.globals(['foo']);
 +        var runner = mocha.run();
 +
 +        var failedTests = [];
 +        runner.on('end', function(){
 +          window.mochaResults = runner.stats;
 +          window.mochaResults.reports = failedTests;
 +        });
 +
 +        runner.on('fail', logFailure);
 +
 +        function logFailure(test, err){
 +
 +          var flattenTitles = function(test){
 +            var titles = [];
 +            while (test.parent.title){
 +              titles.push(test.parent.title);
 +              test = test.parent;
 +            }
 +            return titles.reverse();
 +          };
 +
 +          failedTests.push({name: test.title, result: false, message: err.message, stack: err.stack, titles: flattenTitles(test) });
 +        };
 +      };
 +    </script>

[Jonahss]

Are you still having this issue? Any ideas? Can you give any more information? A link to a job page would be useful.

[jansepar]

Hey @Jonahss, it actually turns out that I had an old API key set, which is why the tests weren't getting kicked off in the first place. Turns out I can still use 4.0.4 to run the tests! I've still had issues with using the latest version of grunt-saucelabs where all of my tests were timing out, but since I haven't had to upgrade I haven't dove too deep.

As for what to do about this ticket - it would have been super helpful to have an error message pop up when I had that old API key. Also if you'd like I could try to upgrade to the latest version of grunt-saucelabs to see why I'm having issues.

[Jonahss]

Yeah, it'd be useful to find out what's going wrong with your tests.

There's definitely better error handling needed. I tried to launch tests today with an unsupported platform and it just hung. I'll create a ticket.

On Tue, Aug 5, 2014 at 11:12 AM, Shawn Jansepar notifications@github.com wrote:

Hey @Jonahss https://github.com/Jonahss, it actually turns out that I had an old API key set, which is why the tests weren't getting kicked off in the first place. Turns out I can still use 4.0.4 to run the tests! I've still had issues with using the latest version of grunt-saucelabs where all of my tests were timing out, but since I haven't had to upgrade I haven't dove too deep.

As for what to do about this ticket - it would have been super helpful to have an error message pop up when I had that old API key. Also if you'd like I could try to upgrade to the latest version of grunt-saucelabs to see why I'm having issues.

— Reply to this email directly or view it on GitHub https://github.com/axemclion/grunt-saucelabs/issues/150#issuecomment-51236873 .

[flowersinthesand]

Yesterday I experienced the same symptom with another reason. It was because I used MAC OSX 10.6.8 where Sauce Tunnel 4.3 can't run (bin/sc is terminated with a single log: 'Illegal Instruction').

FYI, I required the support team to mention each OS's min version and they accepted.

[roblevintennis]

I'm getting this as well. I have a simple hello world qunit, I'm getting success on the CLI but not in the saucelabs console/dashboard (say's Test exceeded maximum duration after 180 seconds):

grunt tests
Running "connect:sauce" (connect) task
Started connect web server on http://localhost:9999

Running "saucelabs-qunit:all" (saucelabs-qunit) task
=> Starting Tunnel to Sauce Labs
>> Connected to Saucelabs

 1 / 3 tests started

 2 / 3 tests started

 3 / 3 tests started

Tested http://localhost:9999/js/tests/index.html
Platform: Windows 8.1,chrome,31
Passed: true
Url https://saucelabs.com/jobs/682002726817400195c4adf7f5294980

Tested http://localhost:9999/js/tests/index.html
Platform: OS X 10.8,safari,6
Passed: true
Url https://saucelabs.com/jobs/c119a8b9f0054d20bb1b399a93c15bd4

Tested http://localhost:9999/js/tests/index.html
Platform: OS X 10.6,firefox,25
Passed: true
Url https://saucelabs.com/jobs/d3f44ec65df04927974ec01bfc6f28e9
>> All tests completed with status true
=> Stopping Tunnel to Sauce Labs

Saucelabs dashboard:

And here I believe I've used the window.global_test_results convention you've instructed to use:

QUnit.done(function (test_results) {
  var tests = [];
  for(var i = 0, len = log.length; i < len; i++) {
    var details = log[i];
    tests.push({
      name: details.name,
      result: details.result,
      expected: details.expected,
      actual: details.actual,
      source: details.source
    });
  }
  test_results.tests = tests;

  window.global_test_results = test_results;
});
QUnit.testStart(function(testDetails){
  QUnit.log(function(details){
    if (!details.result) {
      details.name = testDetails.name;
      log.push(details);
    }
  });
});

test( "hello test", function() {
  ok( 1 == "1", "Passed!" );
});

Deps in package.json:

  "devDependencies": {
    "grunt-autoprefixer": "^0.8.1",
    "grunt-contrib-clean": "^0.6.0",
    "grunt-contrib-connect": "^0.8.0",
    "grunt-contrib-copy": "^0.5.0",
    "grunt-contrib-cssmin": "^0.10.0",
    "grunt-contrib-qunit": "^0.5.2",
    "grunt-contrib-sass": "^0.7.4",
    "grunt-contrib-watch": "^0.6.1",
    "grunt-saucelabs": "^8.3.2"
  }

[jansepar]

Did you add the snippet for onTestComplete? I found that to be causing issues for running tests. I removed that completely and I was able to resolve a lot of issues.

[roblevintennis]

@jansepar I appreciate you chiming in with the suggestion! Unfortunately, I don't think that's my issue:

    /*
    * Saucelabs
    * Requires environment variables set e.g. export SAUCE_USERNAME=XX; export SAUCE_ACCESS_KEY=XX
    *
    */
    'saucelabs-qunit': { //DO NOT CHANGE NAME
      all: {
        options: {
          build: process.env.TRAVIS_JOB_ID,
          concurrency: 3,
          tunnelTimeout: 5,
          urls: ['http://localhost:9999/js/tests/index.html'],
          testname: 'Buttons Sauce Unit Tests',
          browsers: [
            {
              browserName: 'safari',
              version: '6',
              platform: 'OS X 10.8'
            },
            {
              browserName: 'firefox',
              version: '25',
              platform: 'OS X 10.6'
            },
            {
              browserName: 'chrome',
              version: '31',
              platform: 'Windows 8.1'
            }
          ]
        }
      }
    }

So I just set my creds via export before running grunt, but, as you can see no onTestComplete. I haven't done any sort of code diving yet, but it would seem that there's some sort of final handshake that isn't happening to tell saucelabs to consider the test run done. I even looked at one of the videos and I see it run my test in the browser, it passes, and then sort of just sits there and timesout

[Jonahss]

Can you send a link to the job on Saucelabs?

On Sat, Sep 27, 2014 at 6:29 PM, Rob Levin notifications@github.com wrote:

@jansepar https://github.com/jansepar I appreciate you chiming in with the suggestion! Unfortunately, I don't think that's my issue:

/*
* Saucelabs
* Requires environment variables set e.g. export SAUCE_USERNAME=XX; export SAUCE_ACCESS_KEY=XX
*
*/
'saucelabs-qunit': { //DO NOT CHANGE NAME
  all: {
    options: {
      build: process.env.TRAVIS_JOB_ID,
      concurrency: 3,
      tunnelTimeout: 5,
      urls: ['http://localhost:9999/js/tests/index.html'],
      testname: 'Buttons Sauce Unit Tests',
      browsers: [
        {
          browserName: 'safari',
          version: '6',
          platform: 'OS X 10.8'
        },
        {
          browserName: 'firefox',
          version: '25',
          platform: 'OS X 10.6'
        },
        {
          browserName: 'chrome',
          version: '31',
          platform: 'Windows 8.1'
        }
      ]
    }
  }
}

So I just set my creds via export before running grunt, but, as you can see no onTestComplete. I haven't done any sort of code diving yet, but it would seem that there's some sort of final handshake that isn't happening to tell saucelabs to consider the test run done. I even looked at one of the videos and I see it run my test in the browser, it passes, and then sort of just sits there and timesout

— Reply to this email directly or view it on GitHub https://github.com/axemclion/grunt-saucelabs/issues/150#issuecomment-57071530 .

[roblevintennis]

My bad. I played this job: https://saucelabs.com/tests/d3f44ec65df04927974ec01bfc6f28e9

And noticed a failure for runLoggingCallbacks log being undefined in qunit.js

This is called from:

  QUnit.log(function(details){
    if (!details.result) {
      details.name = testDetails.name;

//THIS IS TOP OF ERROR STACK
      log.push(details);
    }
  });

And I resolved it by copying the two lines from you're snippet example which I must have screwed up on and omitted :-1:

var log = [];
var testName;

Now they're passing. Apologies for bogus bug and thanks for the library.

I swear that when I played the video through the other day it was just passing and then hanging. In any event, only thing I suppose could be better is that an error in the test shouldn't result in the Test exceeded maximum duration after 180 error which is a bit misleading.

[Jonahss]

Awesome, glad you got it fixed.

On Mon, Sep 29, 2014 at 1:07 PM, Rob Levin notifications@github.com wrote:

My bad. I played this job: https://saucelabs.com/tests/d3f44ec65df04927974ec01bfc6f28e9

And noticed a failure for runLoggingCallbacks log being undefined in qunit.js

This is called from:

QUnit.log(function(details){ if (!details.result) { details.name = testDetails.name; //THIS IS TOP OF ERROR STACK log.push(details); } });

And I resolved it by copying the two lines from you're snippet example which I must have screwed up on and omitted [image: :-1:]

var log = [];var testName;

Now they're passing. Apologies for bogus bug and thanks for the library.

— Reply to this email directly or view it on GitHub https://github.com/axemclion/grunt-saucelabs/issues/150#issuecomment-57220913 .

[flowersinthesand]

@Jonahss These days I experienced this issue occasionally. As you can see the second log, the test did pass in some browsers. Therefore, I think configuration is not issue. Is there any other cause for this?

https://travis-ci.org/vibe-project/vibe-javascript-client/builds/43790766#L256-L258 https://travis-ci.org/vibe-project/vibe-javascript-client/builds/43791691#L472-L474

Here is saucelabs-mocha config: https://github.com/vibe-project/vibe-javascript-client/blob/82b451937f78d876caaace856852f6bc2b28501f/Gruntfile.js#L13-L59