search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.43k stars 959 forks source link

ArgumentNullException when running benchmarks from published .NET Core app #1018

Closed RealDotNetDave closed 5 years ago

RealDotNetDave commented 5 years ago

I do all of my development in VM's. So to get benchmarks from a non-VM I published the project to a folder and moved it to a different machine. When I run the benchmark via my .NET Core dll, I get the error below:

Unhandled Exception: System.ArgumentNullException: Value cannot be null.
Parameter name: key
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
   at BenchmarkDotNet.Toolchains.CsProj.CsProjClassicNetToolchain.GetCurrentVersion()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at BenchmarkDotNet.Running.BenchmarkRunner.<>c__DisplayClass20_0.<GetSupportedBenchmarks>b__2(BenchmarkCase benchmark)
   at System.Linq.Enumerable.WhereArrayIterator`1.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkRunner.<>c__DisplayClass20_0.<GetSupportedBenchmarks>b__0(BenchmarkRunInfo info)
   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo[] benchmarkRunInfos, IConfig commonSettingsConfig)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo benchmarkRunInfo)
   at dotNetTips.CodePerf.Example.App.Program.Main() in S:\src\VSTS\Rock Your Code - Perf and App Performance\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\Program.cs:line 30
AndreyAkinshin commented 5 years ago

@RealDotNetDave should be fixed now. Could you please try the 0.11.3.932 version from our nightly feed?

RealDotNetDave commented 5 years ago

That did not work (see error below). I get the same thing from the CLR exe too.

// Generate Exception: Unable to find dotNetTips.CodePerf.Example.App in C:\Temp\publish and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj // Build Error: Exception! Message: Unable to find dotNetTips.CodePerf.Example.App in C:\Temp\publish and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj, Stack trace: at BenchmarkDotNet.Toolchains.CsProj.CsProjGenerator.GetProjectFilePath(Type benchmarkTarget, ILogger logger) at BenchmarkDotNet.Toolchains.CsProj.CsProjGenerator.GenerateProject(BuildPartition buildPartition, ArtifactsPaths artifactsPaths, ILogger logger) at BenchmarkDotNet.Toolchains.GeneratorBase.GenerateProject(BuildPartition buildPartition, ILogger logger, String rootArtifactsFolderPath)

adamsitnik commented 5 years ago

When we generate the boilerplate code we need to reference the project that actually defines the benchmarks.

What we do as of today is that we walk the folders down until we find a folder with .sln file. From there, we search for $dllName.csproj including subfolders.

So to run the benchmarks you need to have the source code. If you for some reason can't, you can use InProcessToolchain

RealDotNetDave commented 5 years ago

It seems using the InProcess attribute got it working (tests are still running). But since it does not support Hardward counters and the DisassemblyDiagnoser, I'd really like to get it working without InProcess.

I tried copying the entire source folder to my laptop, I now get these errors:

// Build Error: Standard output: Restoring packages for C:\Temp\Test\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj... Restoring packages for C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj... C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj : error NU1102: Unable to find package BenchmarkDotNet with version (>= 0.11.3.932) C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj : error NU1102: - Found 50 version(s) in nuget.org [ Nearest version: 0.11.3 ] C:\Temp\Test\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj : error NU1102: Unable to find package BenchmarkDotNet with version (>= 0.11.3.932) [C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj] C:\Temp\Test\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj : error NU1102: - Found 50 version(s) in nuget.org [ Nearest version: 0.11.3 ] [C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj] C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj : error NU1102: Unable to find package BenchmarkDotNet.Diagnostics.Windows with version (>= 0.11.3.932) C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj : error NU1102: - Found 23 version(s) in nuget.org [ Nearest version: 0.11.3 ] C:\Temp\Test\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj : error NU1102: Unable to find package BenchmarkDotNet.Diagnostics.Windows with version (>= 0.11.3.932) [C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj] C:\Temp\Test\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj : error NU1102: - Found 23 version(s) in nuget.org [ Nearest version: 0.11.3 ] [C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj] Restore failed in 201.5 ms for C:\Temp\Test\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj. Restore failed in 205.48 ms for C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj.

adamsitnik commented 5 years ago

C:\Temp\Test\dotNetTips.CodePerf.Example.App\publish\cd449890-5bf3-4c26-b975-404063c83a17\BenchmarkDotNet.Autogenerated.csproj : error NU1102: Unable to find package BenchmarkDotNet with version (>= 0.11.3.932)

You forgot to copy the NuGet.config file which tells dotnet restore about where to take the packages from.

RealDotNetDave commented 5 years ago

Ok, copied my Nuget.config file from my VM to my laptop and now seeing the error below. Not sure why it's looking for a package from DevExpress since the test app does not reference it?

// Build Error: Standard output: Restoring packages for C:\Temp\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App.csproj... Restoring packages for C:\Temp\dotNetTips.CodePerf.Example.App\PerfBin\netcoreapp2.2\publish\79057c52-9b26-4fe4-82a1-4e1939bc873c\BenchmarkDotNet.Autogenerated.csproj... C:\Program Files\dotnet\sdk\2.2.101\NuGet.targets(114,5): error : The local source 'p:\Program Files (x86)\DevExpress 18.1\Components\System\Components\Packages' doesn't exist. [C:\Temp\dotNetTips.CodePerf.Example.App\PerfBin\netcoreapp2.2\publish\79057c52-9b26-4fe4-82a1-4e1939bc873c\BenchmarkDotNet.Autogenerated.csproj]

adamsitnik commented 5 years ago

Most probably your global (machine wide) NuGet.config points to this location. To avoid that, you can add a <clear /> element before the feeds:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
    <clear />

    <add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
</configuration>
RealDotNetDave commented 5 years ago

I added that with the same results. I moved the nuget.config to the same folder I'm running the dll from and now and I still see the same error :-(

Here is what my NuGet.config file looks like:

<?xml version="1.0" encoding="utf-8"?>

RealDotNetDave commented 5 years ago

I finally got the benchmark app working, but only after I installed VS Community on my laptop. I'd really like to figure out how to get this running without Visual Studio installed, so I can test on multiple machines.

Also, I tried adding this attribute to my base class: [HardwareCounters(HardwareCounter.BranchInstructions, HardwareCounter.Timer, HardwareCounter.TotalCycles, HardwareCounter.TotalIssues, HardwareCounter.BranchMispredictions, HardwareCounter.CacheMisses)]

But it causes the error below. I don't see that I have a dup in there?

{System.InvalidOperationException: Sequence contains more than one element at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable1 source) at BenchmarkDotNet.Configs.ManualConfig.GetExporters()+MoveNext() at System.Collections.Generic.List1.AddEnumerable(IEnumerable1 enumerable) at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable`1 collection) at BenchmarkDotNet.Configs.ManualConfig.Add(IConfig config) at BenchmarkDotNet.Configs.ManualConfig.Union(IConfig globalConfig, IConfig localConfig) at BenchmarkDotNet.Running.BenchmarkConverter.GetFullConfig(Type type, IConfig config) at BenchmarkDotNet.Running.BenchmarkConverter.TypeToBenchmarks(Type type, IConfig config) at BenchmarkDotNet.Running.BenchmarkRunner.Run[T](IConfig config) at dotNetTips.CodePerf.Example.App.Program.Main() in C:\Temp\dotNetTips.CodePerf.Example.App\dotNetTips.CodePerf.Example.App\Program.cs:line 30} | System.InvalidOperationException

[HardwareCounters(HardwareCounter.BranchInstructions, HardwareCounter.Timer, HardwareCounter.TotalCycles, HardwareCounter.TotalIssues, HardwareCounter.BranchMispredictions, HardwareCounter.CacheMisses)]

adamsitnik commented 5 years ago 
{System.InvalidOperationException: Sequence contains more than one element at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable1 source) at

This issue will be solved when I finish and merge #912 (just letting you know that we are working on it)

AndreyAkinshin commented 5 years ago

@adamsitnik is it resolved now?

adamsitnik commented 5 years ago

@AndreyAkinshin yes, solved by #912