search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.4k stars 957 forks source link

Wrong Measurements and Warnings #2615

Closed swtrse closed 1 month ago

swtrse commented 1 month ago

After Running an Benchmark I got This Warnings at the end

// * Warnings *
ZeroMeasurement
  BenchmarkFlatPrefill10000<UInt256>.SortedList: ShortRun-.NET 9.0      -> The method duration is indistinguishable from the empty method duration
  BenchmarkFlatPrefill10000<UInt256>.SortedList: ShortRun-NativeAOT 8.0 -> The method duration is indistinguishable from the empty method duration
  BenchmarkFlatPrefill10000<UInt256>.SortedList: ShortRun-.NET 8.0      -> The method duration is indistinguishable from the empty method duration

However when looking at the benchmark exports image This are hardly the values of an empty method.

I can provide the whole solution for analysis if that helps just not publicly.

adamsitnik commented 1 month ago

I can provide the whole solution for analysis if that helps just not publicly.

Sharing the log file should be enough. Thanks!

swtrse commented 1 month ago

Just give me a few days I have to rerun the benchmark since I deleted the logs while adding new benchmarks and the benchmark needs roughly 3 days to complete

timcassell commented 1 month ago

3 days?! Can't you just filter the benchmark to repro?

swtrse commented 1 month ago

3 days is the filtered version. It is not my fault that preparation for the benchmark takes this long. The Benchmark itself only takes nanoseconds ;). To be clear I do not now what exactly took so long. All my custom code runs quick (GlobalSetup, IterationSetup and Benchmark), even with debugger hooked. The time is lost somewhere inside BenchmarkDotNet.

timcassell commented 1 month ago

Maybe it's the NativeAOT builds? They can take a very long time. Try removing those. Even then, 3 days is excessive.

swtrse commented 1 month ago

I'll try that. This will at leas half the benchmarks I do run

swtrse commented 1 month ago

Ok. I was unable to reproduce the warning. I will close this issue and open a new one including the log files if I ever run into it again.

AndreyAkinshin commented 2 weeks ago

Theoretically, such situations may emerge in the bimodal case when a significant number of measurements are around zero, but not all of them. It's not easy to handle them properly since we can't distinguish the true bimodality and situations when several measurements are corrupted due to activities by OS or other processes.

I plan to make the logic behind this warning more sophisticated and properly handle such bimodal states.