Provide a working simple example for parameterized tests

[tlestang]

I've been trying to understand how to write parametrized (serial) tests. My strategy was to start from Examples/ParameterizedTest/tests/Test_Cases.pf and change it until it works. Below is a summary of what I had to do to make it compile and run. My main finding is that specifying test parameter values like @test(cases=[1,2,4]) seems to be broken? Note that #256 talks about test prametrization , but does not provide details on how to extend ParameterizedTestCase.

1. The top level module must be named after the filename. In this case this should be test_cases instead of Cases_mod. Using a different name results in a error when linking the test file and the test driver.
2. The constructor for MyParamType is not needed. Only the constructor for the test case type is needed in makeCustomTest.

3. The (generated) suite factory test_cases_suite doesn't compile when specifying the test parameter values directly in the @test decorator.

   ! Generated Test_Cases.F90
   function test_cases_suite() result(suite)
      use FUnit
      use test_cases
      use Wraptest_cases
      implicit none
      type (TestSuite) :: suite
      ! ...
   
      cases = [3,5,7]
      testParameters = [(newMyParam(cases(iCase)), iCase = 1, size(cases))]
   
      do iParam = 1, size(testParameters)
        ! ...

   The counter iCase in the implied do loop defining testParameters is not declared.

   However, declaring the parameter values in the @testcase decorator as the result of a function works well

     ! Test_Cases.pf
     @testCase(testParameters={getParameters()}, constructor=newMyTestCase)
      type, extends (ParameterizedTestCase) :: MyTestCase
         integer :: i
      end type MyTestCase
   
   contains
   
     function getParameters() result(params)
        type(MyParamType), allocatable :: params(:)
        params = [MyParamType(3),MyParamType(5),MyParamType(7)]
      end function getParameters
   
    @test
    subroutine test_even(this)
      ! ...

In the end, I end up with the following test file

module test_cases
   use pfunit
   implicit none

   @testParameter
   type, extends (AbstractTestParameter) :: MyParamType
      integer :: i
   contains
      procedure :: toString
   end type MyParamType

   @testCase(testParameters={getParameters()}, constructor=newMyTestCase)
   type, extends (ParameterizedTestCase) :: MyTestCase
      type(MyParamType) :: param
   end type MyTestCase

contains

   function newMyTestCase(param)
      type (MyTestCase) :: newMyTestCase
      type (MyParamType) :: param
      newMyTestCase%param = param
   end function newMyTestCase

   function getParameters() result(params)
     type(MyParamType), allocatable :: params(:)
     params = [MyParamType(3),MyParamType(5),MyParamType(7)]
   end function getParameters

   @test
   subroutine test_even(this)
      class  (MyTestCase), intent(inout) :: this
      @assertEqual(0, mod(this%param%i,2))
   end subroutine test_even

   function toString(this) result(string)
     ! Unchanged
   end function toString

end module test_cases

Note that in the above I chose to declare MyTestCase with an attribute directly of type MyParam, instead of the underlying integer value.

[tclune]

ParameterizedTestCase is relatively rarely used, so even I will need to go back to look at the source code to familiarize myself with how it is supposed to work. Since there are 2 different "styles" for constructors, it may well be that one has broken and I did not notice as I use the other style. But my intent is for both to work.

This weekend is a 3-day weekend so I should have time to look into this.

[tclune]

The Examples directory in pFUnit is woefully out-of-date. I will delete that in the next release in favor of the pFUnit_demos repository.

I have now added a new directory in the demos for Parameterized tests. It is a fairly simple case that just exercises the basic machinery. I'd like to find a more complex case that is better motivated and yet not overly complex. (I have several real world parameterized tests, but they would be mostly useless for demos.)

Note that I also want to go back and add in the 2nd flavor of parameterized tests. One can parameterize the tests by a simple integer or by a list of test parameters which are a derived type. The current example does the latter, but often the former is a bit easier.

Please check out the updated pFUnit_demos and let me know if it answers your questions. We can then iterate if need be.

[tlestang]

Thanks for looking it up - the demo is very useful.

One thing I'm not sure to undertand is the setUp method

 subroutine setUp(this)
    class (Test_Hypotenuse), intent(inout) :: this

    ! Have to cast the frameworks' testParamater into the subclass
    ! used by the test.  This could be done inside each test, but it
    ! is usually simpler to do it once in the setup.
    select type (p => this%testParameter)
    type is (PythagoreanTriple)
       this%sides = p
    end select
 end subroutine setUp

I'm not sure what this does exactly. Currently I'm still using the test type constructor to attach the parameter type, something like

function newTest_Hypotenuse(testParameter) result(aTest)
      type (Test_Hypotenuse) :: aTest
      class (PythagoreanTriple), intent(in) :: testParameter 

      aTest%sides = testParameter ! attach parameter type to test type
end function newTest_Hypotenuse

what's the difference with the above?

[tclune]

Hmm. Yes - if you have the energy please submit that as a PR to the demos repo.

I think things changed at some point in the past and I so rarely use ParameterizedTest that I've mixed pieces of different styles. Yours avoids the SELECT TYPE and is much preferred!

[tlestang]

Yes - if you have the energy please submit that as a PR to the demos repo.

Very happy to, I should have some time for this tomorrow