search

boost-ext / ut

C++20 μ(micro)/Unit Testing Framework
https://boost-ext.github.io/ut
Boost Software License 1.0
1.26k stars 120 forks source link

Parallel test runner is not actually parallel #641

Open cschreib opened 2 weeks ago

cschreib commented 2 weeks ago

Expected Behavior

The example for the parallel test runner uses std::for_each(std::execution::par, ...) to demonstrate running several test suites in parallel. I expect the tests to actually run in separate threads.

Actual Behavior

In practice on my machine, this isn't actually executed in parallel. As proof: GDB shows no thread being created, and std::this_thread::get_id() is the same in both suites.

If I force the suites to run in separate threads with

    std::thread t1([&]() { suites_[0](); });
    std::thread t2([&]() { suites_[1](); });
    t1.join();
    t2.join();

the console output is interleaving results from both threads. To make this more obvious, I have edited some of assertions to fail in each suite. As you can see, the output is not readable:

test.1.1
test.2.1
test.2.2
test.1.2
test.2.3
test.1.3
Running "test.1
 "test.2.1"...1
  parallel_runner.cpp:64:FAILED [
 "test.1.2"...2 == 1]parallel_runner.cpp:55:FAILED [2 == 1]
FAILED
Running "test.2.3"...PASSED
Running "test.1.3"...PASSED

===============================================================================
tests:   3 | 1 failed
asserts: 6 | 4 passed | 2 failed
.

Steps to Reproduce the Problem

  1. Edit the parallel_runner.cpp example to insert std::cout << "thread ID: " << std::this_thread::get_id() << std::endl; in each test suite.
  2. Compile and run.
  3. Thread IDs are the same.

Specifications