Improving test performance

[Hespian]

We have now reached over 200 tests. Running all of them takes almost 50 seconds on my laptop. This is already bad and it's only going to get worse. Afaik, it is generally considered good practice to have all tests finish in a few seconds (e.g. https://stackoverflow.com/questions/10486/unit-test-execution-speed-how-many-tests-per-second#:~:text=10%20Answers&text=is%20not%20useful-,Show%20activity%20on%20this%20post.,should%20run%20in%201%20second)

Of course in our case tests are slowed down by MPI. But that shouldn't stop us from trying to improve things. For example we could try to not restart the entire MPI runtime environment for each individual unit test.

[niklas-uhl]

When disabling automatic test discovery, the running time is reduced to 15s. The time could reduced even more drastically by linking all tests together and executing the resulting binary only once. GoogleTest still produces useful output and lists each unit test separately, but this is not shown by default when executed using ctest. For now, we can disable automatic test discovery in the CI (but I think we need to introduce a cmake option for that). For the latter approach, I will have a look at it next week

[lukashuebner]

Mine run in 6.5 seconds already … am I missing something or is my laptop that much faster? A factor of 8 seems unlikely, your machine is only 3-4 years older than mine. Which MPI Implementation are you using? :thinking:. Do you have parallelization of the unit tests enabled?

[Hespian]

I'm using OpenMPI 4.1.0 und run tests using make test or ctest . --output-on-failure

In a perfect world, 6.5 seconds would still be pretty slow though ;)

[Hespian]

With ctest . --output-on-failure -j 8 tests take about 9 seconds. That's already much better but I think we should still try to improve it more.

Also, is it possible to configure cmake so that it automatically runs tests in parallel?

[DanielSeemaier]

Also, is it possible to configure cmake so that it automatically runs tests in parallel?

echo "export CTEST_PARALLEL_LEVEL=8" >> ~/.zshrc

[Hespian]

If tests are supposed to be run in parallel, we should probably specify PROCESSORS when registering the test? The documentation even explicitly states that "This is typically used for MPI tests".

[DanielSeemaier]

Since we're just testing functionality, do we really care about that? This sounds like it would make test execution much slower, since it would not run e.g. 3 MPI tests using 4 processes each in parallel on a machine with 8 threads?

[lukashuebner]

I've seen MPI test fail when running with more processes than HW-threads in the past – using ULFM though, maybe this does not happen with a stable MPI implementation.

[Hespian]

@kurpicz has used something to only run tests that have changed since the last time they succeeded. This would at least help locally.

https://github.com/kurpicz/pwm/blob/master/test/util/testsuite.cmake

[Hespian]

Do we plan on doing anything else here or can this be closed?

[Hespian]

Closing this for now.