search

DigitalInBlue / Celero

C++ Benchmark Authoring Library/Framework
Other
824 stars 95 forks source link

Samples and iterations are only computed for first size of problem space (division by zero) #144

Open mloskot opened 4 years ago

mloskot commented 4 years ago

Bug Report

I have a Fixture which returns number of experiments:

    std::vector<celero::TestFixture::ExperimentValue> getExperimentValues() const override
    {
        // Problem space as number of points (pairs of X/Y)
        std::vector<celero::TestFixture::ExperimentValue> v;
        v.emplace_back(1, 0);
        v.emplace_back(256, 0);
        v.emplace_back(512, 0);
        v.emplace_back(1024, 0);
        return v;
    }

The baseline and other benchmarks are defined similarly to this::

BASELINE_F(wkt, to_string, Fixture, 0, 0)
{
    for (auto const& p : this->points_)
    {
        celero::DoNotOptimizeAway(std::to_string(p.x()));
        celero::DoNotOptimizeAway(std::to_string(p.y()));
    }
}

Consider this output, where subsequent sizes of problem space receive unexpected number of iterations, 0 leading to division by zero or similar:

|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  |
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|wkt             | to_string       |               1 |              30 |          262144 |         1.00000 |         0.94479 |      1058432.12 |
|wkt             | to_string       |             256 |              30 |               0 |         1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | to_string       |             512 |              30 |               0 |         1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | to_string       |            1024 |              30 |               0 |         1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | stringstream    |               1 |              30 |          131072 |         1.63591 |         1.54559 |       647000.75 |
|wkt             | stringstream    |             256 |              30 |               0 |        -1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | stringstream    |             512 |              30 |               0 |        -1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | stringstream    |            1024 |              30 |               0 |        -1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | lexical_cast    |               1 |              30 |          131072 |         1.70723 |         1.61298 |       619972.00 |
|wkt             | lexical_cast    |             256 |              30 |               0 |        -1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | lexical_cast    |             512 |              30 |               0 |        -1.00000 |       -nan(ind) |       -nan(ind) |
|wkt             | lexical_cast    |            1024 |              30 |               0 |        -1.00000 |       -nan(ind) |       -nan(ind) |

If I change the getExperimentValues to read

    std::vector<celero::TestFixture::ExperimentValue> getExperimentValues() const override
    {
        std::uint64_t const total_tests = 4;
        // Problem space as number of points (pairs of X/Y)
        std::vector<celero::TestFixture::ExperimentValue> v;
        v.emplace_back(1, uint64_t(std::pow(2, total_tests - 0)));
        v.emplace_back(256, uint64_t(std::pow(2, total_tests - 1)));
        v.emplace_back(512, uint64_t(std::pow(2, total_tests - 2)));
        v.emplace_back(1024, uint64_t(std::pow(2, total_tests - 3)));
        return v;
    }

and I keep the BASELINE_F(wkt, to_string, Fixture, 0, 0), then I get this

|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  |
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|wkt             | to_string       |               1 |              30 |          262144 |         1.00000 |         0.91043 |      1098376.39 |
|wkt             | to_string       |             256 |              30 |               8 |         1.00000 |       249.87500 |         4002.00 |
|wkt             | to_string       |             512 |              30 |               4 |         1.00000 |       489.50000 |         2042.90 |
|wkt             | to_string       |            1024 |              30 |               2 |         1.00000 |       980.00000 |         1020.41 |
|wkt             | stringstream    |               1 |              30 |          262144 |         1.59528 |         1.45240 |       688517.27 |
|wkt             | stringstream    |             256 |              30 |               8 |         1.19510 |       298.62500 |         3348.68 |
|wkt             | stringstream    |             512 |              30 |               4 |         1.21655 |       595.50000 |         1679.26 |
|wkt             | stringstream    |            1024 |              30 |               2 |         1.21939 |      1195.00000 |          836.82 |

where first experiment always gets the number of iterations calculated by Celero, 262144 instead of pre-calculated 16, but subsequent experiments within a set get the iterations from the getExoerimentValues spec.

I'm confused, is the preference of 0 from BASELINE_F for the first experiment an expected behaviour?