dotnet / reactive

The Reactive Extensions for .NET
http://reactivex.io
MIT License
6.68k stars 749 forks source link

Benchmarks don't run for .NET 6+ #2095

Open idg10 opened 6 months ago

idg10 commented 6 months ago

Rx.NET has a benchmark suite, but it currently works only for .NET Framework, and not for .NET 6.0 and later.

If we change the <TargetFramework> to net6.0, the benchmarks fail to run. This seems to be because BenchmarkDotNet expects benchmark projects to use the Release build configuration, but the Rx.NET benchmark project has multiple configurations with the goal of being able to run the benchmarks against older Rx.NET libraries.

This seems like it should offer the following advantages:

That second point is useful if we want to start tracking performance on processor architectures for which we did not have historical data. It's also useful if we want to change the machine type on which we regularly run benchmarks—in addition to being able to re-establish baselines for the current version against new hardware, we can see if the change in performance over time looked different on the new hardware.

Currently it can run benchmarks back as far as v3.1.1.

The problem is that that libraries are selected by choosing different build configurations, and this conflicts with how BenchmarkDotNet wants to do things. (The symptom is a build failure, which I think is caused by inconsistent setting of the CURRENT compilation symbol, meaning that BenchmarkDotNet generates some source code trying to use a benchmark that isn't actually present.)

BenchmarkDotNet now has ways of configuring which external library versions to use. E.g., I found this in a Microsoft blog:

var config = DefaultConfig.Instance
    .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core70)
        .WithNuGet("Microsoft.Extensions.Logging", "7.0.0")
        .AsBaseline())
    .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core80)
        .WithNuGet("Microsoft.Extensions.Logging", "8.0.0-rc.1.23419.4"));

So I think it should be possible for us to use this mechanism to change the library references. (We'd also need to find some way to control the preprocessors defines so that we can omit benchmarks that just can't run on older versions.)