Sometimes, errors and failures during a test aren't caught.

[markehammons]

If you look at my repository here you'll see that my tests are all passing for this commit. However, only 3 tests from BindingSpec were being reported on. After noticing this and investigating, I found that the ZLib test I had written was crashing and stopping the evaluation of the tests following it. This has been a frustratingly common situation with Munit, and I'm not sure if it's because of the nature of my code, or something that needs resolved in Munit.

[SethTisue]

Are you able to construct and provide a self-contained, minimal reproduction of the problem?

[markehammons]

I will give it a shot.

[szymon-rd]

@markehammons Did you manage to find some minimization?

[markehammons]

@szymon-rd I haven't had time these past weeks to try. I will try to do that once I catch up.

[FloWi]

Hi!

I got bitten by the same bug as in the linked issue. I reproduced it in an example project.

https://github.com/FloWi/munit-bug-experiment/blob/main/src/test/scala/example/HelloSpec.scala

In my original project I used Either.catchNonFatal wrong. Apparently it was a fatal exception that I tried to catch in an Either.

This is the output

before assertion in first test
before wrong use of Either.catchNonFatal
example.HelloSpec:
  + first test 0.005s
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 1 s, completed 02.07.2023, 16:34:35

As you can see, the 1st test is displayed correctly. The 2nd test is being executed (the println statement gets written to stdout) but not reported as a failure. It's not displayed at all. The 3rd test doesn't even get executed.

I would prefer mUnit to show the crashed test correctly and not silently hide it.

[FloWi]

It seems that the bug has been introduced in the v1 branch of munit. I tried it with these dependencies

    libraryDependencies ++= Seq(
      "org.typelevel" %% "cats-core" % "2.9.0",
      "org.scalameta" %% "munit" % "0.7.29" % Test,
    )

and it crashed as expected but reports the error.

before assertion in first test
before wrong use of Either.catchNonFatal
example.HelloSpec:
  + first test 0.003s
==> X example.HelloSpec.expected error running tests  0.001s java.lang.IllegalAccessError: asdf
    at example.HelloSpec.$anonfun$new$4(HelloSpec.scala:14)
    at cats.syntax.EitherObjectOps$.catchNonFatal$extension(either.scala:391)
    at example.HelloSpec.$anonfun$new$3(HelloSpec.scala:14)
[error] Test suite example.HelloSpec failed with java.lang.IllegalAccessError: asdf.
[error] This may be due to the ClassLoaderLayeringStrategy (ScalaLibrary) used by your task.

When I use the latest milestone release, it just silently fails.

    libraryDependencies ++= Seq(
      "org.typelevel" %% "cats-core" % "2.9.0",
      "org.scalameta" %% "munit" % "1.0.0-M8" % Test
    )

I hope it helps with the bug-hunt.

[mzuehlke]

Thanks @FloWi for the detailed instruction to reproduce the problem. That made it easy to fix that bug.

[FloWi]

Glad I could help!

Thanks for fixing the bug @mzuehlke!