WebKit / Speedometer

An open source repository for the Speedometer benchmark
Other
592 stars 68 forks source link

Arithmetic mean is displayed in the detailed results instead of geometric mean #407

Closed guillaumenibert closed 3 months ago

guillaumenibert commented 4 months ago

Hello,

By analysing some benchmarks on browserbench.org/Speedometer3.0, I noticed an inconsistency in my results: the value displayed in the detailed results page is an arithmetic mean, not a geometric mean, and the associated deltaPercentage is also related to the arithmetic mean.

Screenshot from 2024-04-16 17-53-49

Looking at the associated JSON, it becomes clear that the displayed value is not the geomean.

...
"Geomean": {
        "name": "Geomean",
        "unit": "ms",
        "mean": 103.97435840648527,
        "geomean": 103.72593600263131,
        "delta": 5.503709786390318,
        "percentDelta": 5.293333732220493,
        "sum": 1039.7435840648527,
        "min": 96.63929714231493,
        "max": 118.13667941264073,
        "values": [
            98.03324073445613,
            99.50013883138098,
            97.04862399436112,
            107.64823016376299,
            103.19284210227842,
            97.74388483491845,
            96.63929714231493,
            107.28020836431752,
            114.52043848442153,
            118.13667941264073
        ],
...

I also double-checked the calculations using the functions in the metric.mjs and statistics.mjs files, the percentDelta value is calculated in relation to the arithmetic mean.

Best regards,

Guillaume Nibert

rniwa commented 4 months ago

Oh, this is a pretty bad bug! Thank you for reporting!

camillobruni commented 4 months ago

I think this is a bit of a confusion in how the data is structured.

In short, we do want to display the mean of all iteration geomeans, e.g.

displayed_geomean = mean(
    geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total),
    ...
    geomean(iterations[N].subtest_1.total, ..., iterations[N].subtest_N.total)
)

I hope I got this somewhat understandable :)

guillaumenibert commented 3 months ago

Yes, I understood that in the JSON, but I think it is a source of confusion, at first glance, a reader of the detailed results will think that Geomean is a geometric mean (of the 10 values). The term Geomean made me think more of a geometric mean than an arithmetic mean. In the formula, I would first have thought of that:

displayed_geomean = geomean(
    geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total),
    ...
    geomean(iterations[N].subtest_1.total, ..., iterations[N].subtest_N.total)
)

and not

displayed_geomean = mean(
    geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total),
    ...
    geomean(iterations[N].subtest_1.total, ..., iterations[N].subtest_N.total)
)

Perhaps displaying "Arithmetic Mean (of geomean values)" as on Speedometer2 rather than "Geomean" in the detailed results page would be easier to understand and formatting the JSON in another way, for example as below:

"Results": {
        "name": "Results",
        "unit": "ms",
        ...
}