Exceptions crash the Test Runners

[maxsnew]

Right now when there's a bug in the code being tested that raises a JS exception such as divide by 0 or a non-exhaustive pattern match, the exception stops the tests from being run. Instead, the test runner should catch such exceptions and report them as failed tests (noting there was an exception).

This can be implemented with a simple Native extension that does the try/catch, but it will only work if the code being run is all thunkified, i.e., we'll want

assertEqual : (() -> a) -> (() -> a) -> Assertion

to be the only interface that people use. This is going to cause a version number bump, but I don't really care about that.

@deadfoxygrandpa any thoughts?

[deadfoxygrandpa]

I agree, now that it's possible to do Native modules I think this library could use some stuff like this.

[deadfoxygrandpa]

I'm starting work on this now.

[deadfoxygrandpa]

@maxsnew Check out the new exceptions branch. The idea you mentioned here is fundamentally working, but there's still work to be done.

[maxsnew]

@deadfoxygrandpa I need to take a closer and test it out a bit but that looks pretty good.

Do you think we should distinguish between failed and errored tests? I'm thinking we may want to report them separately and maybe color them differently in the pretty display (red v grey)?

[deadfoxygrandpa]

I was thinking it might be nice to have a Thunk type type alias Thunk a = () -> a since we're going to be working with so many thunks here. Failed and errored test probably do deserve a different treatment.

[deadfoxygrandpa]

I looked into making a thunk : a -> (() -> a) function, but even with native code there's no way to do it and still wrap exceptions without adding special syntax to the Elm compiler. Adding the syntax is easy enough, and I did it in a local copy of elm-compiler. Do you think something like this might be useful in Elm in general? I'd bring it up on the mailing list, maybe, but it seems odd to add syntax just for a library...

[deadfoxygrandpa]

On that note, I think it's actually impossible to enforce this exception handling without relying on coding style. If you define that thunk function I mentioned, which is a trivial function to write, you can pass the type checking without actually getting any of the benefit that thunkifying everything brings. You need to actually write out a lambda every time.

Not really sure what to think of this. It seems like this branch is adding a lot of indirection without a guarantee of safety.

[deadfoxygrandpa]

This seems like something that could be most easily solved using macros. I know there's been discussions of eventually adding a macro system to Elm itself, but that's probably pretty far off. Another solution would be an Elm-Test preprocessor.