Tests with dynamic data in their name are not executed

[uhla]

Simplified case:

test(s"Test running at ${System.currentTimeMillis()}"){
    fail()
}

When I try to run tests using gradle, this test is not executed when using junit5 platform with scalatest engine (5.1-3.2.18). If I run the same using IntelliJ plugin runner, it's executed without issue.

I suspect this is because the different value there at test registration and then test class execution. We discovered this by accident when we noticed some tests were not running on our CI after update to this library from old junit4 runner and scalatest)

[cheeseng]

For those that is using ScalaTest runner (Intellij plugin included), to run a 'selected' test it is done by passing the -t specifying the test name as the test intended to be run, but in your example, I don't think it is possible for the test name to match what's being passed over.

However for your CI, unless you are using -t in your configuration I don't see why your tests is being skipped.

Having say so, in case you really need to log out the timestamp, you may want to use info (https://www.scalatest.org/scaladoc/1.8/org/scalatest/Informer.html) instead.

[uhla]

I can provide some more details. (I can create a gist with it as well if it helps)

If I have following code:

package com.uhla

class DynamicTest extends AnyFunSuiteLike {

  test(s"(won't run with gradle or intellij single run) this is dynamic value -> ${System.currentTimeMillis()}") {
  }

  test(s"(wwill run with gradle, intellij class run and intellij single run) this is test with static name") {
  }

}

Running in intellij the scala test on the class runs both tests: Run line from intellij:

C:\Users\uhlir\.jdks\corretto-11.0.18\bin\java.exe "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2022.3.2\lib\idea_rt.jar=51627:C:\Program Files\JetBrains\IntelliJ IDEA 2022.3.2\bin" -Dfile.encoding=UTF-8 @C:\Users\uhlir\AppData\Local\Temp\idea_arg_file1631262849 org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner -s com.uhla.DynamicTest -showProgressMessages true

As you correctly pointed out, when I try to run the test itself with dynamic value, it won't run: Run line from intellij:

C:\Users\uhlir\.jdks\corretto-11.0.18\bin\java.exe "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2022.3.2\lib\idea_rt.jar=51504:C:\Program Files\JetBrains\IntelliJ IDEA 2022.3.2\bin" -Dfile.encoding=UTF-8 @C:\Users\uhlir\AppData\Local\Temp\idea_arg_file694659887 org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner -s com.uhla.DynamicTest -testName "(won't run with gradle or intellij single run) this is dynamic value -> ${System.currentTimeMillis()}" -showProgressMessages true

Now when I have this test as part of gradle module, upon running: ./gradlew :myModule:test (or running the test target on module within intellij) test with dynamic value is not executed, but test with static value is executed test-result report shows:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="com.uhla.DynamicTest" tests="1" skipped="0" failures="0" errors="0" timestamp="2024-04-26T14:05:05" hostname="NB1138" time="0.022">
  <properties/>
  <testcase name="(wwill run with gradle, intellij class run and intellij single run) this is test with static name" classname="com.uhla.DynamicTest" time="0.022"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

As far as I know gradle attempts to execute the class, not just a single test using -t. Gradle is setup exactly according to instructions in readme on this github project page. When this test was originally using junit runner for junit4 it got executed even with dynamic value inside the test definition.

Thanks for advice on Informer, I'll look into that as well.

[cheeseng]

@uhla Thanks for the detailed information, I'll try the gradle build as you mentioned and see if I can reproduce the problem.

Thanks!

[cheeseng]

@uhla Fyi I can reproduce your problem with gradlew, I'll try digging further to see what's causing it next.

[cheeseng]

@uhla Fyi I submitted the following PR for this:

https://github.com/scalatest/scalatestplus-junit5/pull/19