Make it easier to write unit tests.

[ta0kira]

For example:

• Basic predicates.
• Some sort of matcher or predicate object.
• An error accumulator. (This might require a new macro to insert the current file+line for a postponed error.)

[ta0kira]

It might also be useful to add a pragma to the compiler to collect a bunch of procedures to execute.

testcase "all tests" {
  success $RunAllTests$
}

concrete Tests {
  @type test1 () -> ()
  @type test2 () -> ()
}

define Tests {
  test1 () { $SingleTest$
    // ...
  }

  test2 () { $SingleTest$
    // ...
  }
}

The semantics are unclear, however. The main point would be to compile them all into a single binary to cut down on compilation time, but dealing with reporting and allowing multiple failures would require some design.

Collection of test procedures would also be a little messy; especially since the ProcedureContext would need to know if the procedure is actually a part of a testcase.

[ta0kira]

An alternative macro/pragma approach:

testcase "all tests" {
  success Tests.run()
}

concrete Tests {
  @type run () -> ()
}

define Tests {
  run () {
    $RunAllTests$
  }

  $UnitTest[test1]$
  test1 () {
    // ...
  }

  $UnitTest[test2]$
  test2 () {
    // ...
  }
}

This way, the $UnitTest$ label won't be visible outside of Tests.

For this to allow multiple failures, however, there will need to be a way to accumulate them, i.e., allow a single test to fail without using fail. This means that there will need to be an additional "failure" macro that logs the error and returns from the test.

Additionally, for that to be feasible within a library (e.g., a regex matcher), there would need to be some sort of exception semantics to short-circuit the rest of the test, so that the caller doesn't need if (!Helpers.test()) { return _ } every time a test helper is called.

Maybe it doesn't need to be full exception semantics; maybe just a special return type could be used.

concrete Helpers {
  @type regexMatch (String,String) -> $TestResult$
}

define Helpers {
  regexMatch (string,regex) {
    if (!Regex.matches(string,regex)) {
      $TestFail[string + " does not match regex " + regex]$
    }
  }
}

Then only allow Helpers.regexMatch to be called in $UnitTest$ or other $TestResult$ functions. Calls to such functions can generate C++ that short-circuits if the call results in a failure. There doesn't necessarily need to even be a return value; it could be done with a ThreadCapture at the top of the $UnitTest$, to avoid boxing and unboxing a bool.

This seems like a hack to sneak in exceptions for unit tests without actually supporting exceptions more generally. The main things that are lacking here are arbitrary return types, user-defined exception types, and the ability to handle exceptions.

[ta0kira]

The overall objective is performance; it takes extremely long to compile the generated C++ for each individual test case, and combined tests (e.g., those in lib/math) currently cannot have multiple failures.

One possible solution for this is to have runSingleTest execute a single test binary multiple times; once per unit test. This is slower than executing every test in one call to the binary, but it will allow tests and helpers to still use fail.

• Since the compiler itself will need to know how many times to call the binary, it will need to know both that it needs to look for multiple tests, and how to target each test when calling the binary. So, rather than success Test.run() in testcase, the testcase should use something like multitest.
• To avoid leaking details to the tests, Argv will need to be handled differently, e.g., by using a static ProgramArgv or by allowing the testcase to specify the args. Note that in the latter case the args can be hard-coded in main; they don't actually need to be passed in.
• Running multiple tests in a single call to runSingleTest would allow zeolite -t to accurately count the individual unit tests, rather than just counting the testcase.

• For simplicity, the function type for each unit test should be @type () -> (), (or similar) just like the main category in a binary.

  If done wrong, this could cause the user a lot of headaches:

  • Specifying the wrong scope, args, returns, etc., and getting an error that basically just says "use @type () -> ()".
  • Needing to use concrete Test {} just to have a place to put all of the test cases, without really needing the instance.
  • Creating multiple concrete categories with test cases, but then the testcase always ignoring all but one of them.

  So, it might make sense to allow something like this at the top level in the testcase:

  unittest foo {  // foo is a function name labeling the test
  // just like any other procedure for () -> (), perhaps with LocalScope
  }

  This wouldn't be dispatched like other functions; it would likely just live directly in test.cpp and get called in main.

To further avoid errors and ambiguities when unittest is used with success Test.run(), maybe success and crash shouldn't have an expression argument. Maybe it should also be an error for a testcase to have no unittest.

[ta0kira]

Closing the loop on the original plan:

• Predicates: lib/testing now has simple comparisons. More can easily be added later.
• Matchers: Interfaces can be added if/when there are tests that need structured object comparisons.
• Accumulators: Not really useful (given the addition of unittest) unless you want to know everything wrong with a particular result (e.g., a parsed document), rather than just the first thing. If that becomes necessary, a matcher should be designed to collect the errors, rather than having a hack to collect "real" errors.