Benchmarks?

[bryceschober]

What would it take to build a benchmark comparison as described in Niall Douglas' ACCU 2017 Talk?

[TartanLlama]

I think having accurate microbenchmarking for something like this is really hard. I don't have time to undertake something like this at the moment, but someone else is welcome to. I imagine that my implementation has very similar performance characteristics to Boost.Outcome.

[jgh-]

Obviously this is two years later but it looks like the Leaf maintainers did a comparison between leaf::result, tl::expected, and boost::outcome here: https://github.com/zajo/leaf/blob/develop/benchmark/benchmark.md

[igormcoelho]

Just happened to know this library, and tried a small benchmark to see if it would replace std::optional on scanner++ project.

I created a simple function that returns an int, std::optional<int> or std::expected<int, E>. Tried this function 10x, 100x and 1000x, using Google Benchmark. Compiler is g++ 9.3 on Ubuntu 20.04, using bazel build system (flags -DNDEBUG -Ofast -fno-exceptions --std=c++17). Results and my interpretations follow below:

---------------------------------------------------------------------------
Benchmark                                 Time             CPU   Iterations
---------------------------------------------------------------------------
bench_direct_int/10                    20.0 ns         20.0 ns     33642583
bench_direct_int/100                    325 ns          325 ns      2139287
bench_direct_int/1000                  2856 ns         2856 ns       231017
bench_optional_int/10                   112 ns          112 ns      6285042
bench_optional_int/100                 1122 ns         1122 ns       623029
bench_optional_int/1000               11230 ns        11228 ns        62272
bench_expected_int/10                  20.4 ns         20.4 ns     34017431
bench_expected_int/100                  273 ns          272 ns      2567835
bench_expected_int/1000                2845 ns         2844 ns       245888
bench_expected_int_int/10               113 ns          113 ns      6263497
bench_expected_int_int/100             1122 ns         1122 ns       623436
bench_expected_int_int/1000           11231 ns        11229 ns        62318
bench_anti_optional_int/10             53.4 ns         53.3 ns     13084551
bench_anti_optional_int/100             533 ns          533 ns      1313907
bench_anti_optional_int/1000           5356 ns         5355 ns       130663
bench_anti_expected_int/10             40.0 ns         40.0 ns     17492936
bench_anti_expected_int/100             342 ns          342 ns      2043960
bench_anti_expected_int/1000           3364 ns         3364 ns       208186
bench_anti_expected_int_int/10         70.1 ns         70.1 ns      9987318
bench_anti_expected_int_int/100         697 ns          697 ns      1001805
bench_anti_expected_int_int/1000       6971 ns         6970 ns       100324

Looks like std::expected<int, std::error_code> takes 2845/2856 ~ 100% (zero overhead) over baseline implementation (without any error handling). It is 4x faster (11230÷2845 ~ 4) than std::optional<int> over its "expected path".

When considering unexpected scenarios: it takes 3364÷2856 ~ 1,17 (around 17% extra overhead) over baseline, and roughly half of the time 3364÷5356 ~ 60% spent by std::optional<int>.

Now, some strange thing (for a newbie like me on the library): when considering tl::expected<int, int>, it gets much slower, even on the "expected path" (11231/11230 ~ 100% roughly the same performance as std::optional<int>). And for unexpected path, it gets 6971÷5356 ~ 1,30 (30% slower than std::optional<int>).

So, for me, it looks like the library is quite optimized for std::error_code, but not that much for other types... anyway, my use-case can consider tl::expected<int, std::error_code> so it's very promising for adoption. Congratulations.

All used files are here: https://github.com/optframe/scannerpp/commit/3bd3cd64da9b3c24da8bf3ffc7231a3e388026d5