Reuse test workers

[Snaipe]

So far, Criterion spawns a whole new process per test. This makes things simple implementation-wise, but it's otherwise not that great on a performance standpoint. execve()s are pretty fast these days, but they still have a non-negligible cost.

An obvious solution here would be to simply have a pool of boxfort workers that each run a bunch of tests sequentially. If a worker dies or exits, we can still tie the failure to the currently executing test, and respawn the worker for more processing.

The -j option would then control the size of the worker pool.

[kugel-]

How will you ensure that each tests starts with a clean, reproducible environment?

Some of our tests play games with signal handlers and symbol interposition.

[kugel-]

Also, do you really do execve()? I thought criterion only fork()s

[Snaipe]

Also, do you really do execve()? I thought criterion only fork()s

Yes, because just fork()ing had some problems.

The first issue was that the network part of criterion was not fork-safe, and trying to make nanomsg fork-safe was pretty-hard. I had to maintain a set of patches to allow that, and even then it was a bit unstable.

The other issue were that anything that the runner was doing in its initialization leaked to the worker process, which caused further issues with standard tools like valgrind or the compiler sanitizers.

[Snaipe]

How will you ensure that each tests starts with a clean, reproducible environment? Some of our tests play games with signal handlers and symbol interposition.

That's a good point. I'm willing to bet that this is not something most tests care about though, so I'd probably be on the side of adding an option to preserve the single-process isolation.

I'm more worried about memory corruption creeping from another test though, but compiling them with address-sanitizer could be a reasonable compromise?