search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.54k stars 969 forks source link

Enable including references in autogenerated csproject #1197

Open billwert opened 5 years ago

billwert commented 5 years ago

Today, if a benchmark needs a reference to a Nuget package it goes into the original project (microbenchmarks.csproj, say) and then does not need to be in the auto generated csproj.

I’d like to enable us to remove the requirement that we run .NET Framework tests on VMs / machines that have Visual Studio installed: https://github.com/dotnet/performance/pull/627. The new Microsoft.NetFramework.ReferenceAssemblies package is precisely the answer.

But it fails because the generated project doesn’t include the package reference I added:

[2019/07/11 19:29:39][INFO] $ dotnet run --project C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net461 --no-restore --no-build -- --anyCategories coreclr corefx --iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --stopOnFirstError true --partition-count 5 --partition-index 0 --packages C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\packages --runtimes net461
[2019/07/11 19:29:42][INFO] // Validating benchmarks:
[2019/07/11 19:29:48][INFO] // ***** BenchmarkRunner: Start   *****
[2019/07/11 19:29:48][INFO] // ***** Found 437 benchmark(s) in total *****
[2019/07/11 19:29:48][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2019/07/11 19:29:50][INFO] // start dotnet restore --packages "C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\packages"  /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 in C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\bin\MicroBenchmarks\Release\net461\2397be0f-2eba-4afc-b8e3-34ec43808e6d
[2019/07/11 19:29:52][INFO] // command took 2.7s and exited with 0
[2019/07/11 19:29:52][INFO] // start dotnet build -c Release  --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 in C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\bin\MicroBenchmarks\Release\net461\2397be0f-2eba-4afc-b8e3-34ec43808e6d
[2019/07/11 19:29:54][INFO] // command took 1.26s and exited with 1
[2019/07/11 19:29:54][INFO] // start dotnet build -c Release  --no-restore --no-dependencies /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 in C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\bin\MicroBenchmarks\Release\net461\2397be0f-2eba-4afc-b8e3-34ec43808e6d
[2019/07/11 19:29:55][INFO] // command took 1.24s and exited with 1
[2019/07/11 19:29:55][INFO] // ***** Done, took 00:00:06 (6.63 sec)   *****
[2019/07/11 19:29:55][INFO] // Found 1 benchmarks:
[2019/07/11 19:29:55][INFO] //   Burgers.Burgers_0: Job-NTJWUA(Runtime=Clr, Toolchain=net461, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)
[2019/07/11 19:29:55][INFO] 
[2019/07/11 19:29:55][INFO] // Build Error: Standard output:
[2019/07/11 19:29:55][INFO] 
[2019/07/11 19:29:55][INFO]  Standard error:
[2019/07/11 19:29:55][INFO]  Time Elapsed 00:00:00.78
[2019/07/11 19:29:55][INFO]     1 Error(s)
[2019/07/11 19:29:55][INFO]     0 Warning(s)
[2019/07/11 19:29:55][INFO] C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\tools\dotnet\x64\sdk\2.1.701\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for framework ".NETFramework,Version=v4.6.1" were not found. To resolve this, install the SDK or Targeting Pack for this framework version or retarget your application to a version of the framework for which you have the SDK or Targeting Pack installed. Note that assemblies will be resolved from the Global Assembly Cache (GAC) and will be used in place of reference assemblies. Therefore your assembly may not be correctly targeted for the framework you intend. [C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\bin\MicroBenchmarks\Release\net461\2397be0f-2eba-4afc-b8e3-34ec43808e6d\BenchmarkDotNet.Autogenerated.csproj]
[2019/07/11 19:29:55][INFO] Build FAILED.
[2019/07/11 19:29:55][INFO] C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\tools\dotnet\x64\sdk\2.1.701\Microsoft.Common.CurrentVersion.targets(1175,5): error MSB3644: The reference assemblies for framework ".NETFramework,Version=v4.6.1" were not found. To resolve this, install the SDK or Targeting Pack for this framework version or retarget your application to a version of the framework for which you have the SDK or Targeting Pack installed. Note that assemblies will be resolved from the Global Assembly Cache (GAC) and will be used in place of reference assemblies. Therefore your assembly may not be correctly targeted for the framework you intend. [C:\dotnetbuild\work\274ec05f-7f6a-45af-b6aa-aa3f40fa2e5d\Payload\artifacts\bin\MicroBenchmarks\Release\net461\2397be0f-2eba-4afc-b8e3-34ec43808e6d\BenchmarkDotNet.Autogenerated.csproj

Interestingly, on my machine the second build (which includes --no-dependencies) works on my machine, probably because it’s using some ambient state at that point.

I see that csproj.txt doesn’t have a parameter for this, so we need some feature to plumb this through.

billwert commented 5 years ago

I thought I could work around this by emitting a Directory.Build.props, but y'all thought of that :) https://github.com/dotnet/BenchmarkDotNet/blob/58fde64c809ceadb3fca9d677a7cec83071b833f/src/BenchmarkDotNet/Templates/CsProj.txt#L4-L5

I definitely understand why this is there, and think it's a good idea. It just prevented the work around I had hoped would be cheap of ensuring that there's a Directory.Build.props in the correct place. As msbuild will stop looking, I can feel confident this is going to have the behavior I wanted. Alas.

adamsitnik commented 5 years ago

This issue is very similar to #1023 :

@viktorhofer @ericstj @eerhardt would it be possible to make Microsoft.NetFramework.ReferenceAssemblies package transitive? or somehow tell MSBuild to reference in C everything that A references? to solve things like #1023 as well?

adamsitnik commented 5 years ago

@billwert if I won't be able to find any clean solution the only idea I currently have it to introduce a concept of BenchmarkDotNet.props files that would be always included by the auto-generated project files.

eerhardt commented 5 years ago

The reason the reference to B isn't included in C is because the reference has PrivateAssets=All on it:

https://github.com/dotnet/performance/pull/627/files#diff-931d1b8d9e5f2892d7f8e001da3bf159R25.

This means that this reference is "private" to A and shouldn't be made transitive.

Possible fixes:

  1. Remove PrivateAssets=All on the reference in the benchmark.csproj (A).
    • I am not sure why marking it as PrivateAssets=All in a benchmark project would be necessary. You don't really expect other projects referencing benchmark projects.
  2. BDN could potentially scan the benchmark .csproj for PrivateAssets=All and copy them to the generated project C.
adamsitnik commented 5 years ago

@eerhardt thank you!

Remove PrivateAssets=All on the reference in the benchmark.csproj

@billwert could you please give it a try?

adamsitnik commented 4 years ago

Remove PrivateAssets=All on the reference in the benchmark.csproj

FWIW this does not solve the problem (tested myself)

kirsan31 commented 2 years ago

I've come here from https://stackoverflow.com/questions/58990324/how-to-benchmark-two-different-versions-of-the-same-non-nuget-library-in-benchma Have the same issue. I need to benchmark some changes in WinForms. For this I manually reference my newly build assembles:

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net6.0-windows</TargetFramework>
        <UseWindowsForms>true</UseWindowsForms>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
    </PropertyGroup>

    <ItemGroup>
        <Reference Include="d:\save\projects\winforms\artifacts\bin\System.Windows.Forms\Release\net6.0\System.Windows.Forms.dll" />
        <Reference Include="d:\save\projects\winforms\artifacts\bin\System.Windows.Forms.Primitives\Release\net6.0\System.Windows.Forms.Primitives.dll" />
    </ItemGroup>

    <ItemGroup>
        <PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
    </ItemGroup>
</Project>

but System.Windows.Forms.dll and System.Windows.Forms.Primitives.dll simply not copied into autogenerated folder - benchmark uses default assemblies :(