timeout_error encountered for (?) fast tests

[sehe]

I wanted to compare to ways of checking whether a boost variant is empty

• http://stackoverflow.com/questions/31320857/how-to-determine-if-a-boostvariant-variable-is-empty/31322574#31322574

However, the which()==0 variant consistently eludes the benchmark algorithm by taking too much time to result in useful measurements. The name of testcase is which and the benchmark results in the following messages:

benchmarking which
which failed to run successfully
benchmark aborted

As you suggested in the lounge:

(And I like that nonius has trouble benchmarking such code because it tells you that even if you coax some results out of it, they might be meaningless)

it might indicate code that's hard to benchmark.

Could you recommend a way to get useful comparison here? Perhaps a rearrangement or a tweak (warmup_time e.g.).

I'd actually be happy if I could clearly see that the test fails to "converge" because it always takes negligible time - because then we have a clear winner if the other variant runs in non-negligible time. I guess in this respect this issue is close to Issue #14

Here's my test program

#include <boost/variant.hpp>
#include <nonius/benchmark.h++>
#include <nonius/main.h++>

namespace detail {
    struct is_blank : boost::static_visitor<bool> {
        constexpr is_blank() = default;

        constexpr bool operator()(boost::blank const&) const { return true; }

        template<typename T>
            constexpr bool operator()(T const&) const { return false; }
    };
}

template <typename... Ts>
constexpr bool is_blank(boost::variant<Ts...> const& v) {
return boost::apply_visitor(detail::is_blank(), v);
}

#include <algorithm>

namespace /*statics*/ {
    // initialization phase not part of benchmarks
    using V = boost::variant<boost::blank, bool, int>;
    static auto const test_data = [] {
        std::vector<V> many;
        std::generate_n(back_inserter(many), 1ul<<12, []()-> V { 
            V r;
            switch (rand()%3) {
                case 1: r = 42;    break;
                case 2: r = false; break;
            }
            return r; // NRVO (not critical)
        });
        return many;
    }();
}

NONIUS_BENCHMARK("apply_visitor", [](nonius::chronometer c) {
    c.measure([&](int i) {
            return is_blank(test_data[ i % test_data.size() ]);
        });
})

NONIUS_BENCHMARK("which", [](nonius::chronometer c) {
    //for (auto & el : test_data) std::cout << (0 == el.which()) << " ";
    //std::cout << '\n';
    c.measure([&](int i) {
            return (test_data[ i % test_data.size() ].which() == 0);
        });
})

Typical output:

$ ./test
clock resolution: mean is 17.7222 ns (81920002 iterations)

benchmarking apply_visitor
collecting 100 samples, 1148 iterations each, in estimated 1722 μs
mean: 13.6744 ns, lb 13.541 ns, ub 14.0019 ns, ci 0.95
std dev: 1.05495 ns, lb 0.508851 ns, ub 1.72528 ns, ci 0.95
found 9 outliers among 100 samples (9%)
variance is severely inflated by outliers

benchmarking which
which failed to run successfully
benchmark aborted