Errors outside of tests don't cause the test run to fail

[bouk]

This seems to have been discussed before in previous issues, but we've ran into multiple problems at Shopify where something in our javascript bundle throw a SyntaxError or ReferenceError or whatever, which would then not make Teaspoon fail and instead cause our CI to silently succeed and us to miss it.

I propose that we make Teaspoon fail the test run if there's an error before Teaspoon.execute is ran, as that will make it non-disrupting when there's errors in asynchronous things, but still makes sure this doesn't silently let errors slip through.

[bouk]

previous issues; https://github.com/modeset/teaspoon/issues/445 https://github.com/modeset/teaspoon/issues/425

[jejacks0n]

Yeah, I've been torn on this because it doesn't indicate a real failure directly. That being said, it's an easy fix with phantomjs, but may not be true elsewhere.

If you decide to look into this I tend to think we might need to do this at the controller level (suite_controller) and handle that exception by ending everything.

---

Jeremy Jackson

On May 5, 2016, at 9:13 AM, Bouke van der Bijl notifications@github.com wrote:

This seems to have been discussed before in previous issues, but we've ran into multiple problems at Shopify where something in our javascript bundle throw a SyntaxError or ReferenceError or whatever, which would then not make Teaspoon fail and instead cause our CI to silently succeed and us to miss it.

I propose that we make Teaspoon fail the test run if there's an error before Teaspoon.execute is ran, as that will make it non-disrupting when there's errors in asynchronous things, but still makes sure this doesn't silently let errors slip through.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub

[bouk]

fixed in #480