Introspective Unit Test

[szaghi]

As a poor Fortraner I was always fascinated by the introspective-power of Python. One of the amazing Python module that I learned to love is doctest. In a few words

The doctest module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown.

For example, in a code like the following

def fib(n):
    """ 
    Calculates the n-th Fibonacci number iteratively 
    >>> fib(0)
    0
    >>> fib(1)
    1
    >>> fib(10) 
    55
    """
    return 0

the comments like >>> fib(0) are VALID UNIT TESTS! When you run the above code with doctest module all the test-comments are tested and their results compared with the expected results (provided immediately after each test in the above example).

During the last night (when Angelica was crying...), I figured out that such a behavior can be mimic by FoBiS.py for Fortran codes (with some limitations).

API desiderata

Let us consider the following Fortran module.

module foo
use bar, only : make_bar
implicit none
private
public :: type_foo, make_foo
type :: type_foo
    !FBStest
    !>>> type(type_foo) :: my_foo
    !>>> call my_foo%init(b=992)
    !>>> call my_foo%print()
    !> 992
    !FBSendtest
    private
    integer :: a
    contains
      procedure, pass(self) :: init
      procedure, pass(self) :: print
  endtype type_foo
  contains
    ! type bound procedures
    subroutine init(self, b)
    class(type_foo), intent(INOUT) :: self
    integer,         intent(IN) ::    b
    self%a = b
    endsubroutine init

    subroutine print(self)
    class(type_foo), intent(IN) :: self
    print*, self%a
    endsubroutine print

    ! non type bound procedures
    function make_foo(b) result(foo_object)
    !FBStest
    !>>> type(type_foo) :: my_foo
    !>>> my_foo = make_foo(b=102)
    !>>> call my_foo%print()
    !> 102
    !FBSendtest
    integer, intent(IN) :: b
    type(type_foo) ::      foo_object
    foo_object%a = b
    endfunction make_foo
endmodule foo

FoBiS.py can:

• build the module with all its dependencies;
• recognize the FoBiS.py doctest, namely the blocks FBStest/FBSendtest;
• for each doctest:
  • create, on the fly, a small volatile program test for each doctest, using the module containing the doctest itself;
  • run the doctest:
  • capture the result printed on stdout and compare with the expected one;
• print a summary of each doctest executed.

  volatile program test skeleton

Fortran is less flexible than Python, than we must use a more verbose syntax for the doctest. In fact, in the above example, the fortran doctest has also the definition of the variable my_foo that is not necessary in the Python doctest. From this point of view the definition of the volatile program test corresponding to each doctest is crucial. In general, a possible skeleton could be:

program volatile_doctest
use foo
implicit none
! user-defined doctest here
endprogram volatile_doctest 

The doctests of the above example are

type bound test

program volatile_doctest
use foo
implicit none
type(type_foo) :: my_foo
call my_foo%init(b=992)
call my_foo%print()
endprogram volatile_doctest 

non type bound test

program volatile_doctest
use foo
implicit none
type(type_foo) :: my_foo
my_foo = make_foo(b=102)
call my_foo%print()
endprogram volatile_doctest 

Limitations

The approach above described should be very easy to implement into the current FoBiS.py version, but the main limitations are:

• the procedures that can be tested are only those defined inside a module;
• we must pass through the standard output for checking the results.

  Considerations

Your thinking, why? The reasons are:

• why not?;
• because I think it is very useful an atomic/micro testing-framework for testing each procedures;
• because it is very nice/elegant/clean/maintainable to embed the test into the procedure docstring;
• because Fortran is better than Python, thus It must have its doctest module :-)

There many limitations (some above pointed-out), nevertheless such a feature is very useful for my OCD programming approach and I will try to implement it soon.

What do you think about?

[jacobwilliams]

I think it's a great idea!

One suggestion: would it be possible to make the syntax more markdown friendly for FORD, so that if you want to have your unit tests syntax-highlightled in the documentation you could do that? Example:

    !FBStest
    !!```Fortran
    !! type(type_foo) :: my_foo
    !! call my_foo%init(b=992)
    !! call my_foo%print()
    !!```
    !!> 992
    !FBSendtest

So, it would grab everything in the Fortran block.

[szaghi]

@jacobwilliams Yes, I cannot test the idea before the next week, nothing is already done, thus the syntax can be modified as we want. Indeed, when this morning I was writing this idea I was already thinking to adopt directly a markdown code-block :-) your suggestion is welcome :-)

[szaghi]

Dear All, I finally finish this feature. Except some limitations (see below) it works incredible well. Monday I will upload a new release (with also some improvements on coarray support). The syntax presently support is the one suggested by jacob (with minor modufication)

    !!```Fortran
    !! type(type_foo) :: my_foo
    !! call my_foo%init(b=992)
    !! call my_foo%print()
    !!```
    !!> 992 <<<

limitations

• only public module-objects can be introspectively tested;
• the test must print the result to stdout.

The second limitation does not hurt me very much, wheras the limitation to only public object make me crazy... I would like to test also private objects. Any suggestions are welcome.

See you monday with this feature.

[szaghi]

Done in the last release. Documentation coming soon.

[szaghi]

See the documentation here.

[zbeekman]

AMAZING! Yes, testing private entities would be quite nice... one quick and dirty trick is to have a wrapping module to provide the public API, that controls/limits the scope of entities it imports.... pretty annoying hack but...