Grouping tests - Githubissues

Many test frameworks support the ability to group tests into a related chunk. This provides multiple benefits:

Real:

Provide the opportunity to perform common setup before asserting a condition
Provide a logical grouping for tests
Have the test results be grouped accordingly

Hypothetical (if the API's design allowed for it)

the ability to configure/enable/disable multiple related tests at once

This is commonly available across language ecosystems! In scalatest we have FunSpec describe/it, WordSpec can/should/in, FreeSpec -/in, RefSpec objects, FeatureSpec Feature/Scenario. Elsewhere, there's the common describe/test in jest (for JS/TS) and elm test, testing/is in clojure and so on.

The first point (common setup) is really key, and is the reason I'll likely stick with scalatest (FreeSpec) for now.

It's much easier to use the standard programming technique of scope to achieve this deduplication, rather than having to adopt a heavyweight tool like Fixtures to try and achieve the same end. It also gives the engineer the flexibility to choose (and change) the resolution of shared setup, instead of being tied to class-files as the minimum 'unit'.

Good testing practice recommends writing one assertion per test, so the ability to share setup like this also creates good incentives for writing nice tests.

In pseudo-code (and using the common choice of describe/test):

describe("my function") {
  describe("when condition <x>") {
    val setupCondition = ???
    test("succeeds in a specific way") {
      val result = myFunction(setupCondition)
      assert(result.property)
    }
    test("succeeds in a different way") {
      val result = myFunction(setupCondition)
      assert(result.anotherProperty)
    }
    test("fails in an expected case") {
      val result = myFunction(setupCondition)
      assert(result.property)
    }
  }

  describe("when condition <y>") {
    ...
  }
}

This example shows tests grouped by a condition, represented as an argument. It could just as easily be some shared state, the selection of a specific test implementation, or a property generator.

Consider an alternative approach where all these assertions happen in a single test, which is presumably what people are doing now with munit. When there's a bug in the implementation, it's much more helpful to be pointed at the exact (sub-)problem by a single specific assertion, with a helpful and targeted test message.

My hunch is that munit started with more of a focus on effectful (perhaps integration-level) tests, where this functionality is less critical. Now that munit's adoption is growing and especially now it is the default selection in the offical Scala project template, I think it would help to implement features that help with unit testing.

Extra optional features might include:

the ability to filter and tag tests by their describe block
customising test settings per describe block (e.g. timeouts, flaky)

Even basic support for this would be great! Ensuring that the test runner and reporter understand this nesting and can use it when conveying success and failure messages would significantly improve munit's utility for unit-testing.

Thank you!

scalameta / munit

Grouping tests #633