Feature Request: Functional Parameters

[12AT7]

My favorite Mettle feature is the template parameters to instantiate tests on different types. However, I have also been experimenting with function parameters to solve a different kind of testing problem. Specifically, I need to sample my tests with different numerical parameters. They are always the same type, and there might be a great many of them. While these parameters can be coaxed into compile time arguments, the Mettle compile times then become prohibitive. Some test patterns are just more naturally expressed via sets of function parameters. This is particularly true in the scientific programming I normally do.

Here I demonstrate a prototype mettle::enumerate<>(...) that is supposed to emulate mettle::subsuite<>(...) as closely as possible, except for these differences:

• It accepts a standard container of parameter values. Actually, it accepts any class that conforms to the range based for interface.
• the name argument for the tests use a libfmt pattern instead of a plain string, and
• The subsuite created is populated by enumerating the container, not by explicit _.test() invocations.

I am completely unmarried to my names here, and I am certain extra wizardry would be required to fully support important Mettle features like fixtures and marks. I am posting this as a feature request, not a fully formed pull request, because I am sure some design discussion up front would be helpful.

From my code sample below, this is the output that I get. I am not actually making any test assertions (those work fine); I am more interested here is seeing which tests are collected and how the names are formatted.

bin/ut.enumerate -o verbose                    (02-15 15:54)
enumerate

  vector<int>
    i=0 PASSED
    i=1 PASSED
    i=3 PASSED
    i=4 PASSED
    i=5 PASSED

  vector<array<int,2>>
    i=0,j=0 PASSED
    i=0,j=1 PASSED
    i=3,j=4 PASSED
    i=5,j=5 PASSED

  vector<tuple<int, float>>
    i=0,j=17.0 PASSED
    i=0,j=42.0 PASSED
    i=3,j=37.0 PASSED

12/12 tests passed

Here is my investigation:

#include <mettle.hpp>
#include <tuple>
#include <functional>
#include <fmt/format.h>

namespace mettle {

// format_index() wraps fmt::format(...) to accept a std::array or std::tuple
// of arguments instead of a varargs list.  Ideally, this would be a function
// of libfmt.
template <typename Tuple>
auto format_index(const std::string format, Tuple&& index) ->
decltype(std::tuple_cat(index), std::string())
{
    return std::apply([format](const auto&... i){ 
            return fmt::format(format, i...); 
            }, std::tuple_cat(index));
}

template <typename T>
auto format_index(const std::string format, T index) ->
typename std::enable_if<std::is_scalar<T>::value, std::string>::type
{
    return fmt::format(format, index);
}

// mettle::enumerate<> is supposed to have similar semantics to
// mettle::subsuite<>, except the parameters are dynamic parameters instead of
// template parameters.  This pattern may be useful in preference to
// subsuite<Args...>, when the number of Args is large, and/or there is no
// useful reason to vary the argument type for each test instance.  The API
// difference is the support of the Generator, which is a container (or generic
// object that obeys the range for interface), to instantiate a subsuite of
// tests, one per element of the range.  The element is very similar in purpose
// to a fixture, but that gets a unique but programmable value for each test.
// The second new argument, "name_format", is a libfmt compatible std::string
// that will serve as the "name" argument of each test.
template <typename Builder, typename Generator, typename Func, typename...Args>
auto enumerate(
        Builder&& builder, 
        std::string subsuite_name,
        std::string name_format,
        const Generator& generator,
        Func&& func
        )
{
    return mettle::subsuite<Args...>(builder, subsuite_name, 
            [name_format, generator, func](auto& _) 
            {
                for (auto idx: generator) {
                    _.test(format_index(name_format, idx), [idx, func]() { func(idx); });
                }
        });
}

} // namespace mettle

mettle::suite<> suite("enumerate", [](auto &_) {

    mettle::enumerate(_, "vector<int>", 
            "i={}", 
            std::vector<int> { 0, 1, 3, 4, 5 },
            [](auto idx) {});

    mettle::enumerate(_, "vector<array<int,2>>", 
            "i={},j={}", 
            std::vector<std::array<int, 2>> { {0,0}, {0,1}, {3,4}, {5,5} },
            [](auto idx) {});

    mettle::enumerate(_, "vector<tuple<int, float>>", 
            "i={},j={}", 
            std::vector<std::tuple<int, float>> { {0,17.0f}, {0,42.0f}, {3,37.0f} },
            [](auto idx) {});

    });

[jimporter]

I'm not sure about this. I intended for this to be handled with a simple for loop or std::for_each or whatever the ranges version of that is. If we lived in a world where there were easy, standard ways to iterate over types, I might never have added support for type-parameterization either.

On the other hand, there's a certain symmetry to providing a special creator for value-parameterized suites. Though on the third(?) hand, "symmetry" is a pretty weak argument.

[12AT7]

The way that this works basically is exactly that; just a for loop over a container (and, presumably, ranges will be similar and awesome). In my use, I actually have a custom generator class that I use as well. It is definitely not required that this function be part of Mettle itself to be useful in applications; it is easy enough to adapt this outside of mettle as I have done.

This is sort of similar to the discussion we had a couple of years ago about supporting properties based testing. It has the similar feel of generating a batch of tests from a programmable rule. It has been a while since I looked at that, but I seem to remember that we settled on a similar pattern of using a free function to populate the suite in a specialized way.

You can close this issue, or use it as an example, or whatever you want to do with it. If others find it useful, then it is still going to be here in issues as a reference.

[jimporter]

I thought about this a bit more, and I think there's some value in doing this, since it's a bit more difficult to parameterize root-level suites with multiple values (obviously, you can't have a for loop at global scope). I'll probably do the same sort of thing for test cases too.