search

elm-explorations / test

Write unit and fuzz tests for Elm code.
https://package.elm-lang.org/packages/elm-explorations/test/latest
BSD 3-Clause "New" or "Revised" License
236 stars 40 forks source link

Show the stack trace in case of caught exceptions #169

Open Janiczek opened 3 years ago

Janiczek commented 3 years ago

For the backstory, refer to https://github.com/mpizenberg/elm-test-rs/issues/90#issuecomment-877514064

TL;DR: I had a faulty Generator in my code, but this message didn't help me much:

It was only after I hacked enough tooling to get to the stack trace that I understood the root cause:

RangeError: Maximum call stack size exceeded
    at Function.f (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:2978:11)
    at A2 (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:57:28)
    at Function.f (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:4681:4)
    at A2 (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:57:28)
    at Function.f (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:4851:25)
    at A3 (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:60:28)
    at shotAndChance (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:7583:18)
    at /Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:2908:10
    at /Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:16:54
    at Function.f (/Users/martinjaniczek/Localhost/lamdera/nu-ashworld/elm-stuff/tests-0.19.1/js/Runner.elm.js:2985:7)

Note the shotAndChance in the middle of the stack trace - that's my function.

I think it would be benefitial if _Test_runThunk also reported the stack trace alongside the message itself, and Test.Runner.run showed that stack trace.

harrysarson commented 3 years ago

I think this might best live behind a flag.

There was a runtime error, run with --backtrace-on-error to see the native backtrack.

Something like that?

gampleman commented 3 years ago

I think we should perhaps take a step back and think of the wider issue. The RangeError: Maximum call stack size exceeded is in my experience almost always due to a faulty generator and/or shrinker. Debugging those is always a difficult experience and while a stack trace might help an experienced developer, it won't be much help to anyone else needing to debug these issues.

So I think we should think about what we can do to help debug either stack overflows in general, or more specifically generator and shrinker problems.

One idea right of the top of my head would be to actually filter and reformat the stack trace. In the rather extreme example above, one could just show something like this:

↓ MyTestModule
↓ MyDescribe
✗ myTestDescription

     Failed by exceeding the maximum stack size

     This can happen when recursive code isn't tail call optimised. Often this can be caused by overly complex fuzzers. 

      We detected the following functions being involved in this error:

              - shotAndChance

     HINT: You can read more about tail call optimisation at <link-to-appropriate-docs>.

i.e. make the error description a little friendlier and filter out most of the noise out of the stack trace:

Janiczek commented 3 years ago

Might be useful: setting the stack trace limit larger than the default 10 lines: https://stackoverflow.com/a/10419288/403702 Supposedly running this in the code does the trick:

Error.stackTraceLimit = Infinity;

Or running node with --stack-trace-limit:

NODE_OPTIONS=--stack-trace-limit=100 node