axemclion / grunt-saucelabs

Grunt task for running all your browser tests using Sauce Labs
MIT License
182 stars 98 forks source link

Fix onTestComplete documentation should pass result.passed instead of !result.passed. #152

Closed ppcano closed 10 years ago

ppcano commented 10 years ago

Readme shows !result.passed which send the wrong value.

Jonahss commented 10 years ago

this is the comment above that example: // The example below negates the result, and also updates the Sauce Labs job's status

So the onTestComplete function in the example, reverses the result of the test. Fail means Pass, Pass means Fail. We actually use this pattern in our test code, to test that a failing test gets reported as a failure.

Since you were misled by the example, can you think of a better way of organizing the README so other's don't get confused?

ppcano commented 10 years ago

@Jonahss, I think, the convention is a little confused.

Do you mean that on the saucelabs reports:

We actually use this pattern in our test code, to test that a failing test gets reported as a failure. I could be missing something but with this behaviour a failing test is reported as Passed.

Anyway, It's ok to follow any pattern. I was just confused and do not clearly understand the reason behind.

Jonahss commented 10 years ago

So normally, you would run your tests without an onTestComplete callback. Everything works fine.

But suppose you wanted some code that ran after each test completed. You can put that code into the onTestComplete callback and intercept the results object. Now you can change the results before they get posted to the console.

In our test suite, we wanted to run a test which failed on Sauce Labs, so make sure we handled failed and error cases correctly. But when the test fails on Sauce, that means we succeeded. So we intercept the results object in the onTestComplete callback and reverse it.

We only have that callback on our failing tests. Check it out here

ppcano commented 10 years ago

@Jonahss, thank you for the detailed info. It's much appreciated.

Jonahss commented 10 years ago

Awesome XD