Choosing a testing framework

[roigcarlo]

@KratosMultiphysics/technical-committee @KratosMultiphysics/altair @KratosMultiphysics/geomechanics

As you know, we have been playing with the idea of moving towards a pure C++ testing framework for some time. We want to push this forward and have it implemented asap. Recently both @huhaas and @roigcarlo have made some drafts (#10718, #10735) to have a closer idea on how that would look and what are the steps we need to follow to implement.

So far we agreed on:

• We want one executable per application (not per test)
• Tests must be 100% independent from python

After some discussion in the @KratosMultiphysics/technical-committee we have come up with a list of what we believe are the most important points on every framework ( we were also considering others but we discarded them)

Since we are not yet reach a consensus on which framework is gonna be used, and we have different points of views on how we should prioritize these list of characteristics (simplicity vs functionality, ease of use vs flexibility, etc...), we would like to collect some feedback from external users (specially from @KratosMultiphysics/geomechanics and @KratosMultiphysics/altair as you may have experience with these) as your points of view on why we should chose one or the other.

From our side, the discussion is mostly between GTest / CTest

GTest

Pros

• We can define custom environments to run
• Have mocks (gmock)
• Wide community
• Documentation
• Usable through CTest

Cons

• It's very heavy
• Migration requires macro changes (more than ~100 files)

DocTest

Pros:

• Lightweight
• Fast compilation time
• Usable through CTest

Cons

• Lacks features (mocks)
• Migration requires macro changes (more than ~100 files)

CTests

Pros

• No library
• Reusing Kratos Home-brew testing mechanism (mostly)
• Usable through CTest
• No changes to macros

Cons

• No features
• Kernel isolation needs to be done manually

[loumalouomega]

@roigcarlo Doctest can be combined with CTest, in fact that was my idea originally, and I have done that in my previous job (here a small example). I certainly recommend the change to CTest as the compilation time is very small, our current framework tests compilations take a significative portion of the compilation time (30% aprox.), changes in respect to current implementation could be minor with "some tricks". Also, as you say Doctest is a very small library, so we will not bloat more the project. I have already some implementations in mind. Other features can be implemented using Kratos testing framework design as base. @philbucher has experience with Doctest as well and is very positive.

Contact me if you want to know the ideas I have in mind.

[roigcarlo]

@loumalouomega We want to know about, it would be useful if you share it here in public if you can.

My main concern regarding DocTest and CTest (I do not speak for the tech committee here, just for myself) is the kernel isolation. I would really really want to see how that looks in the different frameworks, so if you have something in that regard please show us.

[loumalouomega]

Maybe the easiest manner would do a small prototype

[philbucher]

what do you mean by No CMake Runner?

and what about MPI-support? AFAIK only doctest has it

I am using it quite successfully in CoSimIO

[roigcarlo]

what do you mean by No CMake Runner?

If they can be discovered by CMake automatically. I it can I will add it ( we do not have much experience with that one)

and what about MPI-support? AFAIK only doctest has it

Afaik, GTest has suport for MPI as long as you create an environment for it (source):

int main(int argc, char* argv[])
{
  ::testing::InitGoogleTest(&argc, argv);
  ::testing::AddGlobalTestEnvironment(new MPIEnvironment);
  return RUN_ALL_TESTS();
}

+

class MPIEnvironment : public ::testing::Environment
{
public:
  virtual void SetUp() {
    char** argv;
    int argc = 0;
    int mpiError = MPI_Init(&argc, &argv);
    ASSERT_FALSE(mpiError);
  }
  virtual void TearDown() {
    int mpiError = MPI_Finalize();
    ASSERT_FALSE(mpiError);
  }
  virtual ~MPIEnvironment() {}
};

[philbucher]

you mean that you can use ctest?

doctest can do that too: https://github.com/KratosMultiphysics/CoSimIO/blob/master/tests/CMakeLists.txt#L15

[philbucher]

I have some thought on this in general, will comment them in the next days, quite busy rn

[huhaas]

I would pick GTest because it is more feature complete. It has a large community, so more people have prior experience and every question is already asked and answered. Object mocking is very useful (personally a must-have in any test framework) and the scientific check/asserts looks more mature in GoogleTest (float and container comparing). The other test framework does fit the current Kratos. However, in some years I expect we need more of the advanced test features.

The compile-time and runtime improvements sound appealing. But for runtime speed, we are talking about seconds and/or minutes with pure unit tests. Compile time is a valid argument. However, the choice of which test framework will not solve the bigger compile-time problem. The codebase is in general not the fastest to compile. When that is improved the difference in compile speed of the test framework is less important.

From a practical perspective, I would still use CTest as a management layer. Eventually, we end up with more than one test executable. CTest helps with running them and gathering results. It will also make the transition easier. Migrate step by step or application by application.

[philbucher]

My 2 cents:

• we should plan MPI and valgrind (and probably sanitizers) integration right away. Those are key features, that should not be done afterwards. For me this has a super high prio.
• compile times are also very important, especially since a) the compile times are already very long b) tests are enabled by default, even in release, making it quite a pain for new users c) we want to encourage to write as many tests as possible. Compile times should not be blockers here
• I like CTest, it is basic and provides the basic functionalities. I.e. using it with a "proper" testing framework makes sense to me (esp since the valgrind integration works well)
• Fully agree that python must not be part of the C++ tests
• (not a strong opinion) given that we have a lot of macros and porting has proven to be pretty annoying in the past, I would wrap the existing Kratos macros in the test framework macros. E.g. #define KRATOS_CHECK(...) TEST_FRAMEWORK_CHECK_EQUAL. This will make transition much easier
• Regarding advanced features such as mocking. In my experience this has been rarely used in Kratos (in python, where it is part of the standard library) I.e. I am not sure how much this would be used, is it worth the overhead?
• For doctest, what fixtures are missing? Didn't understand the comment
• Having one executable for each app makes full sense. Finally the tests will no longer be in the main libs any more :tada:
• I am missing a bit the planning for the MPI-tests. I guess it would make sense to have one serial and one MPI test-exe per App.

I have only used doctest so far, and I really liked the simplicity and fast compile times. CoSimIO uses doctest, together with valgrind, MPI, different compilers, everything that I consider essential.

I have not used gtest, hence cannot comment on it

[huhaas]

@roigcarlo What are the next steps for this decision? Or does the committee need more information?

[roigcarlo]

We are meeting in an hour to discuss it again

[pooyan-dadvand]

@KratosMultiphysics/technical-committee had a discussion and also invited the @huhaas to know his experience with gtest. In general we tend to GTest do it its complete solution but the decision is postponed to following verifications:

1. Answering the concerns of the @philbucher and @loumalouomega. (Many of them are already answered)
2. Check the compile time using Gtest and verify if the overhead is not too much.

[loumalouomega]

May be interesting to show the reduction time in compilation with Doctest, I have a branch with the transition change, may be interesting to do a fast draft and see how time is reduced.

[rubenzorrilla]

May be interesting to show the reduction time in compilation with Doctest, I have a branch with the transition change, may be interesting to do a fast draft and see how time is reduced.

Just saying that the position is to go for GTest provided that the compilation time stays reasonably similar to current one. My point is to avoid you doing some worthless effort... Having said this, and if I were you, I'd wait until we check the compilation time with GTest. If this gets to be more than what we expect I'd do the test you suggest.

[huhaas]

we should plan MPI and valgrind (and probably sanitizers) integration right away. Those are key features, that should not be done afterwards. For me this has a super high prio.

GTest (and most other test frameworks) take over the main function. It provides argument parsing to select tests, subsets, or individual tests. So, there is no limitation to using Debugging or Valgrind in any way. MPI was answered previously.

Regarding advanced features such as mocking. In my experience this has been rarely used in Kratos (in python, where it is part of the standard library) I.e. I am not sure how much this would be used, is it worth the overhead?

Mocking is one of the most powerful things in software testing. This website (https://www.radview.com/glossary/what-is-mock-testing/) has a nice high-level introduction on why it is useful. It makes it possible to run tests in controllable isolated environments, without having dependencies include dependencies include dependencies etc…

In Kratos this could be used to test Strategies without being dependent on actual Solvers. Or the Model logic can be tested without actual Elements or Conditions. Or test the interaction between Kratos and applications without an actual application. More parts of the Kratos system can be more easily tested and validated. Removing dependencies in tests will also improve the compilation and run speed.

It is true that Kratos is not ready for mock testing yet. However, this is a chicken and egg problem. Having mocking already available will break this cycle.

[loumalouomega]

we should plan MPI and valgrind (and probably sanitizers) integration right away. Those are key features, that should not be done afterwards. For me this has a super high prio.

GTest (and most other test frameworks) take over the main function. It provides argument parsing to select tests, subsets, or individual tests. So, there is no limitation to using Debugging or Valgrind in any way. MPI was answered previously.

Regarding advanced features such as mocking. In my experience this has been rarely used in Kratos (in python, where it is part of the standard library) I.e. I am not sure how much this would be used, is it worth the overhead?

Mocking is one of the most powerful things in software testing. This website (radview.com/glossary/what-is-mock-testing) has a nice high-level introduction on why it is useful. It makes it possible to run tests in controllable isolated environments, without having dependencies include dependencies include dependencies etc…

In Kratos this could be used to test Strategies without being dependent on actual Solvers. Or the Model logic can be tested without actual Elements or Conditions. Or test the interaction between Kratos and applications without an actual application. More parts of the Kratos system can be more easily tested and validated. Removing dependencies in tests will also improve the compilation and run speed.

It is true that Kratos is not ready for mock testing yet. However, this is a chicken and egg problem. Having mocking already available will break this cycle.

For mocking independently of Gtest (and as a small library), there is https://github.com/Znurre/MiniMock, but is LGPL3 which unluckily is a no-go. If mocking is so important, I guess we will need to accept GTest. In any case, I will research if there is any other light and simple mocking lib compatible with BSD.

[huhaas]

DocTest has suggestions on Mock libraries in their FAQ. However, at that point, it becomes a comparison between GTest vs DocTest + Mock Library + Extra Asserts (needed for float, NaN support, and complex container comparing link)

[loumalouomega]

DocTest has suggestions on Mock libraries in their FAQ. However, at that point, it becomes a comparison between GTest vs DocTest + Mock Library + Extra Asserts (needed for float, NaN support, and complex container comparing link)

Good, I'm having seen that. The thing is that maybe we don't need to add everything at once, I mea if using doctest, we can just give support for current tests with the library and in a second phase the support for mocking.

[roigcarlo]

The thing is that maybe we don't need to add everything at once

In this particular case I also lean towards having something more feature complete than lightweight (and you all know I am very picky on big libraries). In short term we will probably not notice the difference between choosing one or the other, but as @huhaas says, people wont start using mocking or any other advanced feature until you give it to them and they see what possible to do or can copy-paste from someone who did it before.

[pooyan-dadvand]

@KratosMultiphysics/technical-committee agrees that GTest is better than Doctest+several other libraries. Now the last point for decision would be:

Comparing the compilation time of the DocTest and GTest. @roigcarlo has branches for testing

[rubenzorrilla]

@roigcarlo can we close this as GTest is already ongoing?

[roigcarlo]

yup