Failing suites in mocha tests now mark saucelabs job as failed

[eelessam]

This is to fix a problem that we had where tests that were failing were marked as passed in SauceLabs.

Given a test that looked like this:

describe('Login: Successful with a verfied user', () => {
    describe('Given I have something', () => {
        it('should be like that', () => {
        });
        describe('When I do something', () => {         
            it('should  be done', () => {
            });
        });
    }); 
});

The first 'describe' line would be used as the title for the test in SauceLabs. This is treat as a suite by the sauce service.

The two 'it's' would be used as a test as they also contain an assertion. If the first 'it' passes but then the test fails in the next describe loop (due to an element not being found etc.), it never then reaches the 2nd 'it'. Which means on the terminal the test has failed but in SauceLabs its marked as passed.

I've added an afterSuite function to check to see if the suite contains an error then to mark the job as failed. Just as the functionality already exists for the type 'feature' in the same file.

[eelessam]

Hi @christian-bromann I'll get this together for you in the next day or two

[christian-bromann]

ping @eelessam

[eelessam]

@christian-bromann

Put together a quick example: https://github.com/eelessam/WDIO-Sauce-Service-Example

If you run the test you'll see if fails but is marked as passed on Saucelabs.

Could you give me a heads up for the other frameworks it needs be working for and I will test. Is it Mocha, Cucumber and Jasmine?

[christian-bromann]

Is it Mocha, Cucumber and Jasmine?

Yes.

Can you clarify the issue. According to https://github.com/eelessam/WDIO-Sauce-Service-Example/blob/master/test/specs/google/failingGoogleTest.spec.js#L14 you actually do something that would let the test fail.

[eelessam]

Yes, the issue is that as you can see I'm making the test fail but on the Saucelabs console the job for this test run is marked as passed.

This is because the sauce-service doesn't deal with or report on the fail that happens in the 'before'. The test obviously doesn't get to the 2nd 'it' so the sauce-service uses the result of the first 'it' as the result that it reports back to SauceLabs.

[christian-bromann]

Published as v0.4.0