kotest / kotest-gradle-plugin

A gradle plugin for Kotest
https://kotest.io
Apache License 2.0
29 stars 5 forks source link

Failing test cases should log their tags #7

Closed jschneidereit closed 3 years ago

jschneidereit commented 3 years ago

tl;dr: kotest should log all tags applied to test cases. This would help with tracking down test case owners (or other test metadata) by allowing kotest consumers to implement custom tags with a helpful toString override.


Consider the concept of multi-team test ownership in a large end-to-end test suite, a way to mark test ownership would be to use Kotest's Tag. Tags as they exist work well as a way to run all tests owned by "team x," but if a job runs the entire test suite and there's a failure then the easiest way to deal with this would be to contact the "owner" of the entire suite not necessarily the owner of the test case.

A way to solve this would be to log the tags applied to each test case, consider the following code:

class SlackTag(override val name: String, private val channel: String) : Tag() {
    override fun toString(): String = "Test owner contact: $channel"
}

val teamB = SlackTag("teamB", "#team-b-channel")

class BTests : ShouldSpec({
    should("do b thing").config(tags = setOf(teamB)) {
        1 shouldBe 2
    }
})

Currently, this results in the following logged message:

BTests > should do b thing FAILED
    org.opentest4j.AssertionFailedError at Owners.kt:25

This is uninformative about who to contact. If we log the tag however we could do something like:

BTests > should do b thing FAILED with tag(s):
  Test owner contact: #team-b-channel
    org.opentest4j.AssertionFailedError at Owners.kt:25

This might be achieved by using clues, but it's a pretty painful thing to add in a clue for each tag, when the tag already exists as metadata on the test case.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.