InvalidMeasurementUnitException when using `--statisticalTest` argument when comparing private builds

[carlossanlop]

I'm trying to run a private runtime build comparison as described here: https://github.com/dotnet/performance/blob/main/src/benchmarks/micro/README.md#private-runtime-builds

The instructions suggest using the argument --statisticalTest 3ms

It throws an exception at the last step of the benchmark comparison because the measurement unit is invalid.

The exception message says I should use ns, so I did that. I got the exact same exception message.

Then I tried to search for other instances and found an old issue mentioning using us, so I did that, and I also got the exact same exception.

What should I use them? Are the instructions and the exception message both suggesting using the wrong unit?

Unhandled exception. Perfolizer.Exceptions.InvalidMeasurementUnitExceptions: Invalid measurement unit: expected ns, but was
   at Perfolizer.Horology.TimeInterval.GetShift(Sample sample)
   at Perfolizer.Metrology.Threshold.<>c__DisplayClass6_0.<EffectiveShift>b__0(IAbsoluteMeasurementValue value)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at System.Linq.Enumerable.DefaultIfEmptyIterator`1.MoveNext()
   at System.Linq.Enumerable.MaxFloat[T](IEnumerable`1 source)
   at Perfolizer.Metrology.Threshold.EffectiveShift(Sample sample)
   at Perfolizer.Mathematics.SignificanceTesting.SimpleEquivalenceTest.Perform(Sample x, Sample y, Threshold threshold, SignificanceLevel alpha)
   at BenchmarkDotNet.Columns.StatisticalTestColumn.GetValue(Summary summary, BenchmarkCase benchmarkCase, Statistics baseline, IReadOnlyDictionary`2 baselineMetrics, Statistics current, IReadOnlyDictionary`2 currentMetrics, Boolean isBaseline)
   at BenchmarkDotNet.Columns.BaselineCustomColumn.GetValue(Summary summary, BenchmarkCase benchmarkCase)
   at BenchmarkDotNet.Columns.BaselineCustomColumn.GetValue(Summary summary, BenchmarkCase benchmarkCase, SummaryStyle style)
   at BenchmarkDotNet.Reports.SummaryTable.<>c__DisplayClass33_2.<.ctor>b__9(IColumn c)
   at System.Linq.Enumerable.SelectArrayIterator`2.Fill(ReadOnlySpan`1 source, Span`1 destination, Func`2 func)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Reports.SummaryTable.<>c__DisplayClass33_1.<.ctor>b__5(BenchmarkReport r)
   at System.Linq.Enumerable.SelectArrayIterator`2.Fill(ReadOnlySpan`1 source, Span`1 destination, Func`2 func)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Reports.SummaryTable..ctor(Summary summary, SummaryStyle style)
   at BenchmarkDotNet.Reports.Summary.GetTable(SummaryStyle style)
   at BenchmarkDotNet.Reports.Summary..ctor(String title, ImmutableArray`1 reports, HostEnvironmentInfo hostEnvironmentInfo, String resultsDirectoryPath, String logFilePath, TimeSpan totalTime, CultureInfo cultureInfo, ImmutableArray`1 validationErrors, ImmutableArray`1 columnHidingRules, SummaryStyle summaryStyle)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, EventProcessor eventProcessor, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, Int32 totalBenchmarkCount, StartedClock& runsChronometer, Int32& benchmarksToRunCount, TaskbarProgress taskbarProgress)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
   at MicroBenchmarks.Program.Main(String[] args) in C:\Users\calope\source\repos\performance\src\benchmarks\micro\Program.cs:line 43

@adamsitnik

[timcassell]

cc @AndreyAkinshin Recently updated the statistical engine.

[AndreyAkinshin]

@carlossanlop thanks for the report. It was a bug I was introduced at the beginning of this week. The fix is in master and v0.13.13-nightly.20240315.146.