User-defined test runner

[data-man]

Usages:

• benchmarks (e.g. if the function name starts with "bench_")
• parallel execution
• testing time
• something other

Possible implementation:

test_runner.zig

const std = @import("std");
const io = std.io;
const builtin = @import("builtin");
const root = @import("root"); //the problem is here: `root` doesn't exist in tests 

pub const io_mode: io.Mode = builtin.test_io_mode;

var log_err_count: usize = 0;

pub fn main() anyerror!void {
    const test_fn_list = builtin.test_functions;

    if (@hasDecl(root, "testRunner")) {
       return root.testRunner(); //the problem is here: `root` doesn't exist in tests 

...///the rest source
}

[FireFox317]

Or maybe this can be solved by having a compiler flag for zig test pointing to the test_runner in use. For example zig test --test-runner ./utils/test_runner.zig

[ityonemo]

I support something like this as it might make test integration less hackish in my "zigler" ffi library.

[demizer]

This would allow me to override the std.log function for my tests.

[data-man]

@demizer Hacked implementation: Sorry, nevermind. It's wrong.

const std = @import("std");
const root = @import("root");

pub fn main() void { //Trick is here!
}

test "root's trick" {
    std.debug.print("main is declared: {}\n", .{ @hasDecl(root, "main") });
}

[PavelVozenilek]

I'd proposed it in #567. Important thing for a custom test runner is ability to have parameters for individual tests and ability to process them by the test runner.

For example:

test (x=10 |10 ms <  time < 100 ms | whatever) // these are test parameters, to be parsed and used by test runner
{
  ...
}

Tests should not be restricted to some special "testing mode". I sometimes rerun tests also in release mode, to make sure release mode didn't screw up something. I also like to run "recent tests" (tests from recently modified source files) every time an application start - it is a handy feature to catch problems really early.

Another useful testing feature would be easy to use mocking: #500.

Parallel execution of tests is not a wise idea. Code not intended for concurrent run would now run in parallel...

[joshgoebel]

So does this mean there is no way to wrap the existing tests at all in any type of infrastructure? Asking with regards to Exercism... we'd need a test runner than can provide JSON output of the individual tests, etc...

Ref: https://github.com/exercism/zig/issues/17

I tried pulling the test_runner.zig directly from 0.8.1 source an compiling it but get errors because the built in's aren't defined:

./lib/test_runner.zig:29:33: error: container 'builtin' has no member called 'test_functions'
    const test_fn_list = builtin.test_functions;
                                ^
./lib/test_runner.zig:10:37: error: container 'builtin' has no member called 'test_io_mode'
pub const io_mode: io.Mode = builtin.test_io_mode;

I tracked this to comp.bin_file.options.is_test in Compilation.zig which evidently decides whether to declare test_io_mode or not, but at this point I'm stuck... I have no idea how I might go about enabling that - and even then the next issue is test_functions and how Zig figures that out auto-magically...

If anyone could provide any advice or is this all way hard-coded and needs an expert to look at it? Obviously the first step would be a custom piece of code (based on test-runner) that has the same "magic" hooked up to it when compiled as zig test... could I do this using zig build?

[ityonemo]

"there is no way" is a bit harsh. "there is no officially supported way" is probably more accurate at the moment. I am currently very much able to wrap zig tests in Zigler package (https://github.com/ityonemo/zigler/) by lexically transforming test "foo bar" {<code>} to fn <some-deterministically-generated-from-"foo bar"-identifier> () !void {<code>} and as far as I understand it that should never not work.

[joshgoebel]

Can you point me to exactly where I should be looking? I skimmed the repo but only saw Elixir, I couldn't find any zig files...

[ityonemo]

yeah, sorry, the transformation I use is kind of buried and in elixir itself, in fact using a hard-to-read-if-you-don't-know-elixir parser combinator library. In any case, the deterministically generated identifier is, i believe "test_". If you are curious about the downstream result of it, it is queued up in this video here: https://www.youtube.com/watch?v=lDfjdGva3NE&t=1440s

[joshgoebel]

Yeah, so might not be immediately helpful to me. My first instinct was to try and modify the existing test runner (vs building tooling from scratch) but it seems very hard wired into the mechanics. All I really need is to change the output method, etc.

[matu3ba]

If anyone wants to work on this, stage1 has the following "offensive function", which hardcodes the test_runner path:

static ZigPackage *create_test_runner_pkg(CodeGen *g) {
    return codegen_create_package(g, buf_ptr(g->zig_std_special_dir), "test_runner.zig", "std.special");
}

Also pub fn create in Compilation.zig is called 3 times, which is an interesting behavior that can be observed with the following changes to print the respective strings:

```zig diff --git a/src/Compilation.zig b/src/Compilation.zig index bd7581863..45c7fd842 100644 --- a/src/Compilation.zig +++ b/src/Compilation.zig @@ -1341,12 +1341,29 @@ pub fn create(gpa: Allocator, options: InitOptions) !*Compilation { ); errdefer std_pkg.destroy(gpa); + const stdout = std.io.getStdOut().writer(); + try stdout.writeAll("directory: "); + try stdout.writeAll(options.zig_lib_directory.path.?); + try stdout.writeAll("\n"); + try stdout.writeAll("root_src_dir_path: "); + try stdout.writeAll("std" ++ std.fs.path.sep_str ++ "special"); + try stdout.writeAll("\n"); + try stdout.writeAll("root_src_path: "); + try stdout.writeAll("test_runner2.zig"); + try stdout.writeAll("\n"); + try stdout.writeAll("\n"); + + // printed 3 times: + // directory: /home/user/dev/git/zig/zig/testrunner/lib + // root_src_dir_path: std/special + // root_src_path: test_runner.zig + const root_pkg = if (options.is_test) root_pkg: { const test_pkg = try Package.createWithDir( gpa, options.zig_lib_directory, "std" ++ std.fs.path.sep_str ++ "special", - "test_runner.zig", + "test_runner2.zig", ); errdefer test_pkg.destroy(gpa); ```

[matu3ba]

It looks like the name of the test runner ("test_runner2.zig") must be in sync with the name used in codegen_create_package:

            const root_pkg = if (options.is_test) root_pkg: {
                const test_pkg = try Package.createWithDir(
                    gpa,
                    options.zig_lib_directory,
                    "std" ++ std.fs.path.sep_str ++ "special",
                    "test_runner2.zig",
                );
                errdefer test_pkg.destroy(gpa);

                try test_pkg.add(gpa, "builtin", builtin_pkg);
                try test_pkg.add(gpa, "root", test_pkg);
                try test_pkg.add(gpa, "std", std_pkg);

                break :root_pkg test_pkg;
            } else main_pkg;
            errdefer if (options.is_test) root_pkg.destroy(gpa);

If one is deviating, compilation of the test fails. I do see 3 ways to solve this:

1. replicate the functionality in stage1 and stage2 which means to modify argument parsing in int main(int argc, char **argv) in zig0.cpp and using the respective test commands
2. write the stuff in C to use it from both stage1 and stage2
3. figure out how to prevent stage1 to segfault on a modified test_runner argument

[matu3ba]

More of an WIP idea after discussion at work, for which this issue is the closest thing I could find.

Identified use cases of mocks in C and C++:

• 1. workarounds due to no error definitions to have at least some test coverage of error conditions
• 2. count function calls (and sequences of executions) as poor-man minified execution trace to not being forced to enumerate the possible/necessary states for a partial runtime trace (which can be painful for some standards/business logic)

The first is not needed, because we have explicit error semantics.

The second requires either to either 1. have some function/runtime tracer lib for counting logic on traces or 2. comptime extension of containers + comptime wrappers for function calls and (if necessary) linker scripts to substitute the functions. I know, very hacky. However, it should be (very) useful to test network stuff or anything to eliminate/abstract slow Kernel calls.

Alternatives ?

Related use case: Helpers to create test case from object state (ie from gdb) and a fast way to make zig code from it to lazy-create a test case. Generating with from gdb snippets and use lsp to generate an objects instance and fill it with values would be leet.