elm-explorations / test

Write unit and fuzz tests for Elm code.
https://package.elm-lang.org/packages/elm-explorations/test/latest
BSD 3-Clause "New" or "Revised" License
236 stars 40 forks source link

Shrink runtime exception failures #221

Open Janiczek opened 1 year ago

Janiczek commented 1 year ago

Currently when a test throws a runtime exception, we catch that and return a test failure. (Good.) But we don't shrink the input.

This should be doable: instead of

try {

  do {
    input = fuzz()
    test result = runTest(input)
  } while test result is not failure and attempt count limit is not reached 

  if test result is failure {
    test result = shrink(test result)
  }

  return test result

} catch {
  return failure
}

we could do:

do {
  input = fuzz()

  try {
    test result = runTest(input)
  } catch {
    test result = failure
  }

} while test result is not failure and attempt count limit is not reached

if test result is failure {
  test result = shrink(test result)
}

return test result

so that crash failures behave the same way as other failures, with "Given $INPUT:" reported, that input properly shrunk, etc.