ta0kira / zeolite

Zeolite is a statically-typed, general-purpose programming language.
Apache License 2.0
18 stars 0 forks source link

Add `lib/testing` support for custom test matchers #207

Closed ta0kira closed 10 months ago

ta0kira commented 11 months ago

Inspired by Catch2 (v3) matchers, but better.

Starting requirements:

  1. Composable, e.g.,

    \ fooMap `Matches:with` MapMatcher:from(HashedMap<String,FooMatcher>.new()
       .set("foo1",FooMatcher.from(foo1))
       .set("foo2",FooMatcher.from(foo2))
       .set("foo3",FooMatcher.from(foo3)))
    
    // ...
    
    define KVMatcher {
     [KVReader<#x,#y>&DefaultOrder<KeyValue<#x,#y>>] expected
    
     // ...
    
     compare (state,actual) (state2) {
       state2 <- state
       traverse (expected.defaultOrder() -> KeyValue<#x,#y> expectedCurrent) {
         if (! `present` state2) {
           break
         }
         optional FooMatcher actualCurrent <- actual.get(expectedCurrent.getKey())
         // do something push expectedCurrent.getKey() to the state scope
         state2 <- state2 `expectedCurrent.compare` actualCurrent
         // do something to pop expectedCurrent.getKey() to the state scope
       }
       // check for extra values
     }
    }
    
    // ...
    
    // maybe also allow short-circuiting
    // (See #206 for `&.`.)
    
    return state
       &.check(value1,matcher1)
       &.check(value2,matcher2)
       &.check(value3,matcher3)
  2. Custom matchers testable without requiring a crash.

  3. When used in a test, caller can choose between failing now vs. later.

  4. When composed in a matcher, matcher can end early or collect many errors, independently of top-level caller's choice.

  5. Supports nested scope descriptions and hierarchical summary output.


Thoughts:

  1. The only part of this specific to testing is the failure part, which means that this could probably generalized into something useful outside of tests.
  2. Not sure how to "fail later" when called in a test.
    1. Caller can't return anything.
    2. Whatever object is tracking the test won't know when to do the final check, and the test writer could forget to execute the check at the end.
    3. What if there are multiple separate objects tracking state in a single test?
    4. Could be done in a C++ destructor, but that won't work with a different reference strategy.
ta0kira commented 11 months ago

Maybe a new $TestsOnly$ built-in:

concrete Testcase {
  // This is cleaner than a singleton when it comes to calling functions.
  @category thisTest () -> (Testcase)

  // Each check would add itself upon construction.
  // - AsBool returns false if the check failed.
  // - Builder<Formatted> generates the report if/when needed.
  @value addCheck<#x>
    #x requires AsBool
    #x requires Builder<Formatted>
  (#x reporter:) -> (#x)

  // Used the same way fail is used in tests now, but formats the full report first.
  // - Collects all failed checks.
  // - Prints report.
  // - Exits _without_ a crash/stack trace.
  @value failNow () -> ()

  // Called in main after the unittest returns.
  // - Same as failNow but only fails if there are failed checks.
  @value checkNow () -> ()
}

The main problem is that there isn't going to be a stack trace for errors to lead someone back to the failed line.

  1. Perhaps easily solvable with a macro $CallTrace$ with type optional Order<Formatted>, although that could lead to some weird meta-programming, e.g., different behavior when called from certain files. Maybe that could be defined only in $TestsOnly$ contexts?

  2. addCheck could easily grab the trace where it was called from, but that'll just lead to a factory function. Unless those functions were $NoTrace$.

    define Matches {
     with (value,matcher) { $NoTrace$
       SomeTestState state <- SomeTestState.new()
       // addCheck grabs the trace, which leads to the Matches:with call.
       Testcase:thisTest().addCheck(state)
       \ state `matcher.compare` value
     }
    }
  3. Maybe not a great solution, but a macro could be expanded to capture the context.

    unittest test {
     $OptionalCheck[ 1 `Testing.equals` 2 ]$
     $RequiredCheck[ 2 `Testing.equals` 3 ]$
    }

    The main problem is that someone might forget to use it, but maybe they could use OptionalCheck to postpone the failure.

ta0kira commented 11 months ago

I think this would be a better approach:

// In base
@type interface Testcase {
  start () -> ()
  finish () -> ()
}
// In lib/testing
concrete TestChecker {
  defines Testcase

  // ...
}

define TestChecker {
  @category Bool testing <- false

  start () {
    if (testing) {
      fail("Test already in progress.")
    }
    testing <- true
  }

  finish () {
    if (!testing) {
      return _
    }
    testing <- false
    // check stuff...
  }
}
// In foo.0rt
testcase "something" {
  success TestChecker  // Optional. Anything that defines Testcase allowed.
}

unittest test {  // <- calls TestChecker.start()
  // ...
}                // <- calls TestChecker.finish()

An additional consideration is an exit(0) call inadvertently dismissing pending test errors. To solve this, we could have a ThreadCapture that registers a cleanup to get executed when exit is called, then add a call to TestChecker.finish().

ta0kira commented 10 months ago

Another interesting idea:

\ value `Matches:with` (Sequence:contains("foo") `Matches:or` Sequence:contains("bar"))
\ value `Matches:with` `Matches:not` Sequence:contains("baz")

Maybe the logic could be based on example/parser, e.g., having some sort of recoverable error that allows a reset. The main caveat would be that the object being tested might not behave immutably during testing.

ta0kira commented 10 months ago

Some matcher ideas:

  1. Ignore whitespace and/or case. (lib/util)
  2. Within epsilon for AsFloat. (lib/math)
  3. Map/set matches/intersects expected. (lib/container)
ta0kira commented 10 months ago

I realized that for visibility reasons some objects need to implement their own matching. This effectively means that all matchers need to be non-$TestsOnly$, although we can still make evaluation $TestsOnly$.

ta0kira commented 10 months ago

As far as what can be supported, I think there are 2 main scenarios:

  1. Container objects with some param #x that might not define Equals<#x>. In this case, the container will generally provide a way to access the values, so a ValueMatcher<Foo<#x>> could be created.
  2. Opaque objects that either don't use params or can include #x defines Equals<#x>, etc. In this case, the object can implement TestCompare<Foo<#x>>. The caveat is that params might need #x requires TestCompare<#x> if they don't define Equals<#x>.

There are some situations that don't fit into one of the above, but those already had testability issues.