search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.24k stars 952 forks source link

BenchmarkDotNet test adapter not running tests with dotnet test #2570

Closed sheffer21 closed 2 months ago

sheffer21 commented 2 months ago

I am encountering an issue when trying to run BenchmarkDotNet tests using the "dotnet test -c release" command. Despite having the BenchmarkDotNet test adapter configured, no tests are being executed. When running the project tests from Visual Studio all the tests are discovered and executed

Steps to Reproduce:

  1. Create a new .NET project.
  2. Add BenchmarkDotNet.TestAdapter package.
  3. Define some benchmark tests.
  4. Run "dotnet test -c release" in the project directory.

Expected Behavior: The benchmark tests should be discovered and executed

Actual Behavior: No tests are executed. The output indicates that zero tests were run.

Environment: BenchmarkDotNet.TestAdapter version: 0.13.12 Microsoft.NET.Test.Sdk: 17.8.0

Root Cause: -> When running "dotnet test" the VsTestAdapter is running "VsTestAdapter.RunTests(IEnumerable? sources, IRunContext? runContext, IFrameworkHandle? frameworkHandle)" https://github.com/dotnet/BenchmarkDotNet/blob/205ce61313e7f1b1410d72b20951736b54c70cfd/src/BenchmarkDotNet.TestAdapter/VSTestAdapter.cs#L82 -> The "RunTests" function calls "VsTestAdapter.RunBenchmarks(string source, IFrameworkHandle frameworkHandle, IEnumerable? testCases = null)" with testCases set to null https://github.com/dotnet/BenchmarkDotNet/blob/205ce61313e7f1b1410d72b20951736b54c70cfd/src/BenchmarkDotNet.TestAdapter/VSTestAdapter.cs#L71 -> Calls "BenchmarkExecutorWrapper.RunBenchmarks" with caseIds as null https://github.com/dotnet/BenchmarkDotNet/blob/205ce61313e7f1b1410d72b20951736b54c70cfd/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkExecutorWrapper.cs#L13 -> Calls "BenchmarkExecutor.RunBenchmarks" with benchmarkIds as null -> The executor filters all the tests because "benchmarkIds" is null https://github.com/dotnet/BenchmarkDotNet/blob/205ce61313e7f1b1410d72b20951736b54c70cfd/src/BenchmarkDotNet.TestAdapter/BenchmarkExecutor.cs#L41 -> After filtering all the tests it returns without executing anything: https://github.com/dotnet/BenchmarkDotNet/blob/205ce61313e7f1b1410d72b20951736b54c70cfd/src/BenchmarkDotNet.TestAdapter/BenchmarkExecutor.cs#L56-L57