Closed guillaumenibert closed 3 months ago
Oh, this is a pretty bad bug! Thank you for reporting!
I think this is a bit of a confusion in how the data is structured.
Geomean
(uppercase) refers to metric which contains the geomean of the subtest times.values
contains an entry per iteration so that's values[0] = geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total)
geomean
is the geomean of all the Geomean
values
(this is mostly auto-added for completeness but doesn't add much value in this context)mean
is the geomean of all the Geomean
values
(this is what's displayed)In short, we do want to display the mean of all iteration geomeans, e.g.
displayed_geomean = mean(
geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total),
...
geomean(iterations[N].subtest_1.total, ..., iterations[N].subtest_N.total)
)
I hope I got this somewhat understandable :)
Yes, I understood that in the JSON, but I think it is a source of confusion, at first glance, a reader of the detailed results will think that Geomean is a geometric mean (of the 10 values). The term Geomean made me think more of a geometric mean than an arithmetic mean. In the formula, I would first have thought of that:
displayed_geomean = geomean(
geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total),
...
geomean(iterations[N].subtest_1.total, ..., iterations[N].subtest_N.total)
)
and not
displayed_geomean = mean(
geomean(iterations[0].subtest_1.total, ..., iterations[0].subtest_N.total),
...
geomean(iterations[N].subtest_1.total, ..., iterations[N].subtest_N.total)
)
Perhaps displaying "Arithmetic Mean (of geomean values)" as on Speedometer2 rather than "Geomean" in the detailed results page would be easier to understand and formatting the JSON in another way, for example as below:
"Results": {
"name": "Results",
"unit": "ms",
...
}
Hello,
By analysing some benchmarks on browserbench.org/Speedometer3.0, I noticed an inconsistency in my results: the value displayed in the detailed results page is an arithmetic mean, not a geometric mean, and the associated deltaPercentage is also related to the arithmetic mean.
Looking at the associated JSON, it becomes clear that the displayed value is not the geomean.
I also double-checked the calculations using the functions in the metric.mjs and statistics.mjs files, the percentDelta value is calculated in relation to the arithmetic mean.
Best regards,
Guillaume Nibert