search

jejacks0n / teaspoon

Teaspoon: Javascript test runner for Rails. Use Selenium, BrowserStack, or PhantomJS.
1.43k stars 243 forks source link

Error reporting swallowed when running tests. #336

Closed theotherdon closed 9 years ago

theotherdon commented 9 years ago

First off, thanks for all of your work on this gem. It makes front-end testing with Rails really easy to pick up.

I seem to be running into an error reporting issue when running my tests. I'm working on an Angular / Rails app and have successfully setup and run several tests with Teaspoon. All was well until I decided to refactor my tests to dry them up. Teaspoon started reporting 0 examples, 0 failures despite having valid (or so I thought) test files in place. After some digging, I figured out that I had a syntax error in one of my javascript test files. I figured this out by running Mocha manually on the test file and Mocha properly reported the error which allowed me to identify the syntax error and successfully rerun the test. I then tried running the test in Teaspoon and it was back to working normally.

I know that I can lint the javascript files to catch stuff like this, but I think it would be a good idea to have Teaspoon report any errors that prevent it from running tests.

This issue seems to affect both Jasmine and Mocha. I can provide more detailed instructions on reproducing the issue, but there's not really much you have to do to get it there. Just stick some kind of syntax error in one of your javascript test files and you should see it.

jejacks0n commented 9 years ago

Thanks for the kind words.

The exception should show up in your console if you're running teaspoon in the browser -- you have to check for these, and I don't have a better solution for this because a parse error comes long before teaspoon can do much -- as far as I understand parse errors to be handled.

Additionally, if you were to run it from the command line it should display an error there, though I know that there's maybe a conflict with the better_errors gem. Hope that helps, and let me know what you end up finding -- this has been a pretty consistent issue, but based on how browsers work, and how better_errors tries to take everything over at the rack level I don't have a good solution for the potential issues that can be present.

mikepack commented 9 years ago

This is a known problem in Mocha: https://github.com/mochajs/mocha/issues/553

jejacks0n commented 9 years ago

This isn't quite the same @mikepack -- I think what's being described here is this is an exception at parse time, which halts all further javascript execution, and so Teaspoon can't do anything at that point really, other than try to override window.onerror, but that seems like an area we probably want to avoid.

jejacks0n commented 9 years ago

What you're thinking about @mikepack is the ability to toggle the try/catch button so exceptions provide a more useful stack trace, and one where execution can be halted for debugging purposes -- instead of masking them behind a try/catch that may only provide you vague insight into what might be happening to cause the error.

theotherdon commented 9 years ago

Sorry for the delay in getting back to you.

That makes sense. I though it would report errors when being run from the console as well. However, the behavior was the same both in the browser and in the console.

I haven't had much of a chance to dig into it, but I'll let you know what I find when I am able to get to it.