Cannot run explicitly tagged tests only using kotest

[Rattlehead931]

Hi, I have a maven project where I have both junit5 and kotest test classes. But I whenever I try to run a test using a specific tag, all the tests end up running. I have submitted a question on SO with bounty here. Reaching out here because I did not get any answers yet. Can anyone help me with this?

My pom looks like below

<properties>
    <kotlin.version>1.7.0</kotlin.version>
    ...
   </<properties>
   <dependencies>
    <dependency>
        <groupId>io.kotest</groupId>
        <artifactId>kotest-runner-junit5-jvm</artifactId>
        <version>5.3.1</version>
        <scope>test</scope>
    </dependency>
      ...
   </dependencies>
     .....
   <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.0.0-M6</version>
            <configuration>
                <forkCount>1</forkCount>
                <reuseForks>true</reuseForks>
                <runOrder>alphabetical</runOrder>
                <useUnlimitedThreads>true</useUnlimitedThreads>
                <redirectTestOutputToFile>true</redirectTestOutputToFile>
                <argLine>-Xmx2560m -Dspring.test.context.cache.maxSize=24 --enable-preview ${argLine}</argLine>
            </configuration>
    </plugin>

Test class looks something like below

@Tags("CustomTest")
class CustomTestClass : FunSpec({
   blah blah
})

The command I am using to try and run the tests in the class is like below

./mvnw test -Dkotest.tags="CustomTest"

[OliverO2]

Could you check whether the system property kotest.tags="CustomTest" is actually passed from the command line to the forked surefire JVM? You can try changing the argline setting in your POM to:

<argLine>-Dkotest.tags=CustomTest -Xmx2560m -Dspring.test.context.cache.maxSize=24 --enable-preview ${argLine}</argLine>

[Rattlehead931]

I don't think its being passed. Because hardcoding it in the argline is still running other tests

[Rattlehead931]

Curiously I ran prgep java to get the PIDs of the spawned JVM processes. Then used jcmd PID VM.system_properties to check the properties. All of them had the kotest.tags=Custom property set to correct value.

[OliverO2]

Initially, you wrote kotest.tags=CustomTest, the latest post says kotest.tags=Custom.

Can you provide a minimal reproducer?

[OliverO2]

You could also run the test with debug logging enabled like this:

env KOTEST_DEBUG=true ./mvnw test -Dkotest.tags="CustomTest"

Then you should find a file kotest.log. Look inside for a line like this (shortened here). What does it show after "Active tags:"?

6535740  TestEngine              TestEngine: Active tags:

[Rattlehead931]

Then you should find a file kotest.log. Look inside for a line like this (shortened here). What does it show after "Active tags:"?

That was a typo sorry. The property was set to CustomTest

env KOTEST_DEBUG=true ./mvnw test -Dkotest.tags="CustomTest" running this command throws an error. Shows an error like below

org.junit.platform.commons.JUnitException: TestEngine with ID 'kotest' failed to discover tests with an additional error Caused by: java.io.FileNotFoundException: /home/sam/development/workspace/kotest/kotest/kotest.log (No such file or directory)

This is not my username in the machine. It seems like the path to the logfile is hardcoded somehow

[OliverO2]

That's unfortunately hardcoded in your Kotest version (this has been corrected since). Could you temporarily create the directory and look at the log file there?

[Rattlehead931]

I am on macos. I cannot create a directory in /home. But I tried creating a structure like /sam/development/workspace/kotest/kotest/kotest.log under my Users/username/ directory. But the error messages are still the same. Any other suggestions?

[Rattlehead931]

I updated to 5.6 and got a more workable error message which expects the log file to be here /Users/emil/workspace/kotest/kotest but in the generated log I couldn't find below pattern.

TestEngine: Active tags:

There was no instance of Active tags or tags in that file

[OliverO2]

The "Active tags" log entry has been in Kotest since 2021. Kotest 5.6.0 is of April 2023. Please post the contents of the log file from the first line up to (including) a line containing the words "Engine started".

[Rattlehead931]

I let the command run for its entirety and I was able to find the Active tags: CustomTest in the log. Here is the portion of the log file you requested https://pastebin.com/5tQT6XMy

[OliverO2]

I get

Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.

Could you enable public access?

[Rattlehead931]

I get

Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.

Could you enable public access?

Done

[OliverO2]

OK, so we know the system property is in fact present in the Kotest invocation.

Assuming that only some of your tests are tagged with "CustomTag", Kotest would initially consider all tests (Maven Surefire would invoke it that way), then filter out the tests which are not tagged with "CustomTag".

I am using Gradle, not Maven Surefire. This means that test output may look differently when we compare experiences.

You wrote:

all the tests end up running

Can you provide excerpts of the output from your test run

• for one test tagged with "CustomTest"
• for one test not tagged with "CustomTest"?