search

kotest / kotest-intellij-plugin

The official Kotest plugin for Intellij and Android Studio
Apache License 2.0
179 stars 25 forks source link

Test names containing line breaks are ignored when run individually #292

Open sndawson opened 7 months ago

sndawson commented 7 months ago

This may mostly be a Kotest issue rather than a plugin issue, but I think there is something the plugin could do to work around this.

Running plugin version 1.3.74-IC-2023.1.

The problem appears to have been introduced in Kotest 5.0.0 from my testing.

I have seen the issue with Kotest versions 5.0.0, 5.5.5 and 5.81, Kotlin versions 1.6, 1.7 and 1.9 as well as JVM 11 and 17. The issue does not occur with Kotest version 4.6.4.

An example file:

import io.kotest.core.spec.style.BehaviorSpec

class MainTests : BehaviorSpec({
    Given("If you run from this Given, the test will run") {
        println("Given test 1")
        When(
            """
            But if you run from this When
            with line breaks,
            the test will be ignored
            """
        ) {
            println("When test 1")
            Then("It will also be ignored if you try to run it from here") {
                println("Then test 1")
            }
        }
    }
    Given("""A test case
        with a line break
        at a Given will also be ignored
        """
    ) {
        println("Given test 2")
        When("And same if you try to run it from here") {
            println("When test 2")
            Then("Same here") {
                println("Then test 2")
            }
        }
    }
    Given("""Test names with line breaks
        that trim any part of the string
        don't even get the run button
    """.trimMargin()) {
        println("Given test 3")
        When("And this test will be ignored even if you run it from here") {
            println("When test 3")
            Then("Same at this spot") {
                println("Then test 3")
            }
        }
    }
    Given("""Line
        breaks
        """
    ) {
        println("Given test 4")
        When("Shorter test name") {
            println("When test 4")
            Then("Shorter test name also") {
                println("Then test 4")
            }
        }
    }
})

Expected behaviour (as seen in 4.6.4) (the individual test runs): image

Behaviour with kotest version 5.X (all tests are ignored): image the output:

C:\Users\sdawson\.jdks\corretto-17.0.10\bin\java.exe -Didea.test.cyclic.buffer.size=1048576 "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2023.1.2\lib\idea_rt.jar=60675:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2023.1.2\bin" -Dfile.encoding=UTF-8 @C:\Users\sdawson\AppData\Local\Temp\idea_arg_file1953867495 io.kotest.engine.launcher.MainKt --spec MainTests --testpath "A test case
        with a line break
        at a Given will also be ignored
        " --reporter teamcity
Testing started at 7:30 PM ...

If you run from this Given, the test will run is excluded by test filter(s): Excluded by test path filter: 'A test case
        with a line break
        at a Given will also be ignored
        '

A test case with a line break at a Given will also be ignored is excluded by test filter(s): Excluded by test path filter: 'A test case
        with a line break
        at a Given will also be ignored
        '

Test names with line breaks that trim any part of the string don't even get the run button is excluded by test filter(s): Excluded by test path filter: 'A test case
        with a line break
        at a Given will also be ignored
        '

Line breaks is excluded by test filter(s): Excluded by test path filter: 'A test case
        with a line break
        at a Given will also be ignored
        '

Process finished with exit code 0



I can raise a Kotest issue as well, but I think the plugin could get around this issue by removing line breaks from the test name. For example, if I run the 'Line \n breaks' test from above without the 'Shorten command line' option, the plugin runs a long command ending with this:

io.kotest.engine.launcher.MainKt --spec MainTests --testpath "Line
        breaks
        " --reporter teamcity

If I copy that command, remove the line break and then run it myself, kotest does successfully run the test:

io.kotest.engine.launcher.MainKt --spec MainTests --testpath "Line breaks" --report

Output:

##teamcity[testSuiteStarted name='MainTests' id='MainTests' locationHint='kotest:class://MainTests:1']
Given test 4
When test 4
Then test 4
##teamcity[testIgnored name='Given: If you run from this Given, the test will run' id='MainTests/If you run from this Given, the test will run' parent_id='MainTests' locationHint='kotest:file://MainTests.kt:4' message='If you run from this Given, the test will run is excluded by test filter: Exclude(reason=Excluded by test path filter: |'Line breaks|')' result_status='Ignored']
##teamcity[testIgnored name='Given: A test case with a line break at a Given will also be ignored' id='MainTests/A test case with a line break at a Given will also be ignored' parent_id='MainTests' locationHint='kotest:file://MainTests.kt:19' message='A test case with a line break at a Given will also be ignored is excluded by test filter: Exclude(reason=Excluded by test path filter: |'Line breaks|')' result_status='Ignored']
##teamcity[testIgnored name='Given: Test names with line breaks that trim any part of the string don|'t even get the run button' id='MainTests/Test names with line breaks that trim any part of the string don|'t even get the run button' parent_id='MainTests' locationHint='kotest:file://MainTests.kt:32' message='Test names with line breaks that trim any part of the string don|'t even get the run button is excluded by test filter: Exclude(reason=Excluded by test path filter: |'Line breaks|')' result_status='Ignored']
##teamcity[testSuiteStarted name='Given: Line breaks' id='MainTests/Line breaks' parent_id='MainTests' locationHint='kotest:file://MainTests.kt:44']
##teamcity[testSuiteStarted name='When: Shorter test name' id='MainTests/Line breaks -- Shorter test name' parent_id='MainTests/Line breaks' locationHint='kotest:file://MainTests.kt:49']
##teamcity[testStarted name='Then: Shorter test name also' id='MainTests/Line breaks -- Shorter test name -- Shorter test name also' parent_id='MainTests/Line breaks -- Shorter test name' locationHint='kotest:file://MainTests.kt:51']
##teamcity[testFinished name='Then: Shorter test name also' id='MainTests/Line breaks -- Shorter test name -- Shorter test name also' parent_id='MainTests/Line breaks -- Shorter test name' duration='2' locationHint='kotest:file://MainTests.kt:51' result_status='Success']
##teamcity[testSuiteFinished name='When: Shorter test name' id='MainTests/Line breaks -- Shorter test name' parent_id='MainTests/Line breaks' duration='7' locationHint='kotest:file://MainTests.kt:49' result_status='Success']
##teamcity[testSuiteFinished name='Given: Line breaks' id='MainTests/Line breaks' parent_id='MainTests' duration='26' locationHint='kotest:file://MainTests.kt:44' result_status='Success']
##teamcity[testSuiteFinished name='MainTests' id='MainTests' locationHint='kotest:class://MainTests:1']