`asClue` and `withClue` do not print clues when the block throws an exception that is not of type `AssertionFailedError`

[kirillzh]

Kotest version: 5.9.0

Consider test:

test("foo") {
  "some clue".asClue {
    listOf("a", "b")
      .single { it.length == 2 }
      // chain some other assertion/operation on returned element
  }
}

On failure a clue is not printed:

List is empty.
java.util.NoSuchElementException: List is empty.
    at kotlin.collections.CollectionsKt___CollectionsKt.single(_Collections.kt:616)

Expected something like:

some clue
java.util.NoSuchElementException: List is empty.
    at kotlin.collections.CollectionsKt___CollectionsKt.single(_Collections.kt:616)

I suspect this is because asClue and withClue only work when the block throws kotest's AssertionFailedError. And because single throws kotlin's NPE or IllegalArgumentException, it doesn't work nicely with clues.

In many cases, our tests and testing infrastructure performs assertions without using kotest APIs (using methods like single, first, etc that throw non AssertionFailedError exceptions). Because of this, majority of our test failures are missing clues.

It would be very helpful if asClue/withClue worked still when any exception is throws.

Slack kotlinlang discussion: https://kotlinlang.slack.com/archives/CT0G9SD7Z/p1726702539652919

[OliverO2]

For clue support, it is not the type of exception that matters. Kotest actually relies on inserting the clue message at throw time, which it does via this function:

fun failure(message: String, cause: Throwable?): AssertionError {
   return Exceptions.createAssertionError(clueContextAsString() + message, cause)
}

To work around this restriction, you could use something like this withClues function (note the plural form).

But I wonder whether we could switch to make augmenting exceptions by intermediate catch blocks the default in Kotest. We'd need to look at how that influences stack traces. Also, clues could look a bit nicer and have their message appear visually separated.