Closed markehammons closed 1 year ago
Are you able to construct and provide a self-contained, minimal reproduction of the problem?
I will give it a shot.
@markehammons Did you manage to find some minimization?
@szymon-rd I haven't had time these past weeks to try. I will try to do that once I catch up.
Hi!
I got bitten by the same bug as in the linked issue. I reproduced it in an example project.
https://github.com/FloWi/munit-bug-experiment/blob/main/src/test/scala/example/HelloSpec.scala
In my original project I used Either.catchNonFatal
wrong. Apparently it was a fatal exception that I tried to catch in an Either.
This is the output
before assertion in first test
before wrong use of Either.catchNonFatal
example.HelloSpec:
+ first test 0.005s
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 1 s, completed 02.07.2023, 16:34:35
As you can see, the 1st test is displayed correctly. The 2nd test is being executed (the println statement gets written to stdout) but not reported as a failure. It's not displayed at all. The 3rd test doesn't even get executed.
I would prefer mUnit to show the crashed test correctly and not silently hide it.
It seems that the bug has been introduced in the v1 branch of munit. I tried it with these dependencies
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "2.9.0",
"org.scalameta" %% "munit" % "0.7.29" % Test,
)
and it crashed as expected but reports the error.
before assertion in first test
before wrong use of Either.catchNonFatal
example.HelloSpec:
+ first test 0.003s
==> X example.HelloSpec.expected error running tests 0.001s java.lang.IllegalAccessError: asdf
at example.HelloSpec.$anonfun$new$4(HelloSpec.scala:14)
at cats.syntax.EitherObjectOps$.catchNonFatal$extension(either.scala:391)
at example.HelloSpec.$anonfun$new$3(HelloSpec.scala:14)
[error] Test suite example.HelloSpec failed with java.lang.IllegalAccessError: asdf.
[error] This may be due to the ClassLoaderLayeringStrategy (ScalaLibrary) used by your task.
When I use the latest milestone release, it just silently fails.
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "2.9.0",
"org.scalameta" %% "munit" % "1.0.0-M8" % Test
)
I hope it helps with the bug-hunt.
Thanks @FloWi for the detailed instruction to reproduce the problem. That made it easy to fix that bug.
Glad I could help!
Thanks for fixing the bug @mzuehlke!
If you look at my repository here you'll see that my tests are all passing for this commit. However, only 3 tests from BindingSpec were being reported on. After noticing this and investigating, I found that the ZLib test I had written was crashing and stopping the evaluation of the tests following it. This has been a frustratingly common situation with Munit, and I'm not sure if it's because of the nature of my code, or something that needs resolved in Munit.