tinylibs / tinybench

🔎 A simple, tiny and lightweight benchmarking library!
MIT License
1.73k stars 36 forks source link

Variance is computed incorrectly #44

Closed bagdonas closed 1 year ago

bagdonas commented 1 year ago

Hey, noticed you compute variance incorrectly (and in turn other stats that depend on it).

https://github.com/tinylibs/tinybench/blob/a0e3c9cfd95962b49888ddb8271e38c25e5645ac/src/utils.ts#LL18C57-L18C57

reduce isn't initialized with a value (2nd argument) so first sample get used as initial sum and iteration starts with the second sample. Depending on sample duration this can dramatically affect the relative mean error. A simple fix:

-  const result = samples.reduce((sum, n) => sum + ((n - mean) ** 2));
+  const result = samples.reduce((sum, n) => sum + ((n - mean) ** 2), 0);

Cheers

Aslemammad commented 1 year ago

Oh, thank you so much, nice catch! you want to create a PR?

bagdonas commented 1 year ago

Sure, there you go.