dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.5k stars 965 forks source link

Wrong results report when comma is used as decimal separator #2003

Closed alex-jitbit closed 2 years ago

alex-jitbit commented 2 years ago

On my Windows system I have comma as a decimal separator.

When running a benchmark these are the results I get:

// * Detailed results *
BenchMark.Test1: ShortRun(IterationCount=3, LaunchCount=1, WarmupCount=3)
Runtime = .NET 6.0.4 (6.0.422.16404), X64 RyuJIT; GC = Concurrent Workstation
Mean = 1.163 us, StdErr = 0.006 us (0.50%), N = 3, StdDev = 0.010 us
Min = 1.156 us, Q1 = 1.157 us, Median = 1.158 us, Q3 = 1.166 us, Max = 1.174 us
IQR = 0.009 us, LowerFence = 1.143 us, UpperFence = 1.180 us
ConfidenceInterval = [0.980 us; 1.346 us] (CI 99.9%), Margin = 0.183 us (15.73% of Mean)
Skewness = 0.36, Kurtosis = 0.67, MValue = 2
-------------------- Histogram --------------------
[1.147 us ; 1.184 us) | @@@
---------------------------------------------------

BenchMark.Test2: ShortRun(IterationCount=3, LaunchCount=1, WarmupCount=3)
Runtime = .NET 6.0.4 (6.0.422.16404), X64 RyuJIT; GC = Concurrent Workstation
Mean = 831.283 ns, StdErr = 1.463 ns (0.18%), N = 3, StdDev = 2.534 ns
Min = 829.238 ns, Q1 = 829.866 ns, Median = 830.494 ns, Q3 = 832.306 ns, Max = 834.118 ns
IQR = 2.440 ns, LowerFence = 826.205 ns, UpperFence = 835.967 ns
ConfidenceInterval = [785.049 ns; 877.517 ns] (CI 99.9%), Margin = 46.234 ns (5.56% of Mean)
Skewness = 0.28, Kurtosis = 0.67, MValue = 2
-------------------- Histogram --------------------
[826.931 ns ; 836.425 ns) | @@@
---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19044.1645 (21H2)
Intel Core i7-9700K CPU 3.60GHz (Coffee Lake), 1 CPU, 8 logical and 8 physical cores
.NET SDK=6.0.202
  [Host]   : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT
  ShortRun : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT

Job=ShortRun  IterationCount=3  LaunchCount=1
WarmupCount=3

| Method |       Mean |     Error |   StdDev |
|------- |-----------:|----------:|---------:|
|  Test1 | 1,162.9 ns | 182.91 ns | 10.03 ns |
|  Test2 |   831.3 ns |  46.23 ns |  2.53 ns |

As you can see in the "detailed results" the first test took 1.184 ns, the second took 826.931 ns.

But in the summary it looks like the second test IS FASTER

alex-jitbit commented 2 years ago

sorry confused us and ns. please disregard