Closed ta0kira closed 10 months ago
Maybe a new $TestsOnly$
built-in:
concrete Testcase {
// This is cleaner than a singleton when it comes to calling functions.
@category thisTest () -> (Testcase)
// Each check would add itself upon construction.
// - AsBool returns false if the check failed.
// - Builder<Formatted> generates the report if/when needed.
@value addCheck<#x>
#x requires AsBool
#x requires Builder<Formatted>
(#x reporter:) -> (#x)
// Used the same way fail is used in tests now, but formats the full report first.
// - Collects all failed checks.
// - Prints report.
// - Exits _without_ a crash/stack trace.
@value failNow () -> ()
// Called in main after the unittest returns.
// - Same as failNow but only fails if there are failed checks.
@value checkNow () -> ()
}
The main problem is that there isn't going to be a stack trace for errors to lead someone back to the failed line.
Perhaps easily solvable with a macro $CallTrace$
with type optional Order<Formatted>
, although that could lead to some weird meta-programming, e.g., different behavior when called from certain files. Maybe that could be defined only in $TestsOnly$
contexts?
addCheck
could easily grab the trace where it was called from, but that'll just lead to a factory function. Unless those functions were $NoTrace$
.
define Matches {
with (value,matcher) { $NoTrace$
SomeTestState state <- SomeTestState.new()
// addCheck grabs the trace, which leads to the Matches:with call.
Testcase:thisTest().addCheck(state)
\ state `matcher.compare` value
}
}
Maybe not a great solution, but a macro could be expanded to capture the context.
unittest test {
$OptionalCheck[ 1 `Testing.equals` 2 ]$
$RequiredCheck[ 2 `Testing.equals` 3 ]$
}
The main problem is that someone might forget to use it, but maybe they could use OptionalCheck
to postpone the failure.
I think this would be a better approach:
// In base
@type interface Testcase {
start () -> ()
finish () -> ()
}
// In lib/testing
concrete TestChecker {
defines Testcase
// ...
}
define TestChecker {
@category Bool testing <- false
start () {
if (testing) {
fail("Test already in progress.")
}
testing <- true
}
finish () {
if (!testing) {
return _
}
testing <- false
// check stuff...
}
}
// In foo.0rt
testcase "something" {
success TestChecker // Optional. Anything that defines Testcase allowed.
}
unittest test { // <- calls TestChecker.start()
// ...
} // <- calls TestChecker.finish()
An additional consideration is an exit(0)
call inadvertently dismissing pending test errors. To solve this, we could have a ThreadCapture
that registers a cleanup to get executed when exit
is called, then add a call to TestChecker.finish()
.
Another interesting idea:
\ value `Matches:with` (Sequence:contains("foo") `Matches:or` Sequence:contains("bar"))
\ value `Matches:with` `Matches:not` Sequence:contains("baz")
Maybe the logic could be based on example/parser
, e.g., having some sort of recoverable error that allows a reset. The main caveat would be that the object being tested might not behave immutably during testing.
Some matcher ideas:
lib/util
)AsFloat
. (lib/math
)lib/container
)I realized that for visibility reasons some objects need to implement their own matching. This effectively means that all matchers need to be non-$TestsOnly$
, although we can still make evaluation $TestsOnly$
.
As far as what can be supported, I think there are 2 main scenarios:
#x
that might not define Equals<#x>
. In this case, the container will generally provide a way to access the values, so a ValueMatcher<Foo<#x>>
could be created.#x defines Equals<#x>
, etc. In this case, the object can implement TestCompare<Foo<#x>>
. The caveat is that params might need #x requires TestCompare<#x>
if they don't define Equals<#x>
.There are some situations that don't fit into one of the above, but those already had testability issues.
Inspired by Catch2 (v3) matchers, but better.
Starting requirements:
Composable, e.g.,
Custom matchers testable without requiring a crash.
When used in a test, caller can choose between failing now vs. later.
When composed in a matcher, matcher can end early or collect many errors, independently of top-level caller's choice.
Supports nested scope descriptions and hierarchical summary output.
Thoughts: