Allow the users to compare 32 vs 64 RyuJit for .NET Core

[adamsitnik]

As of now if the default .NET Sdk is configured to x86, then the user can run only x86 benchmarks. So no 32 vs 64 with single config ;(

I have reported the issue at dotnet cli repo.

The only idea I have now is to ask the user to provide the custom path for dotnet cli. But I don't want to do it that way..

[AndreyAkinshin]

An idea about the "x86 vs x32" .NET Core problem: we can detect all available dotnet.exe (e.g. via where dotnet, it should be available on all modern versions of Windows). Next, we can check the architecture of each dotnext.exe (e.g. like this, it works perfectly for C:\Program Files\dotnet\dotnet.exe and C:\Program Files (x86)\dotnet\dotnet.exe on my Windows 10). Finally, we can use dotnet.exe which is matched to the user config (or print a message that says something like "sorry, we didn't find appropriate runtime"). @adamsitnik, what do you think?

[adamsitnik]

I would like to hear more from the dotnet cli team first. I created issue for that. If this problem can not be solved on dotnet cli level then we can try your idea or we could introduce similar param to what we did for Mono (custom path).

I had exactly the same problem with dnx, but with dnx it was possible to use dvvm to specify the architecture.

[adamsitnik]

Done.

Users are supposed to provide custom dotnet cli path. I did not implement any discovery because it would be hard to do it right and the gain would be very small.

Sample config:

[Config(typeof(CustomPathsConfig))]
[DisassemblyDiagnoser]
public class Jit_RyuJitX86
{
    public class CustomPathsConfig : ManualConfig
    {
        public CustomPathsConfig() 
        {
            var dotnetCli32bit = NetCoreAppSettings
                .NetCoreApp20
                .WithCustomDotNetCliPath(@"C:\Program Files (x86)\dotnet\dotnet.exe", "32 bit cli");

            var dotnetCli64bit = NetCoreAppSettings
                .NetCoreApp20
                .WithCustomDotNetCliPath(@"C:\Program Files\dotnet\dotnet.exe", "64 bit cli");

            Add(Job.RyuJitX86.With(CsProjCoreToolchain.From(dotnetCli32bit)).WithId("32 bit cli"));
            Add(Job.RyuJitX64.With(CsProjCoreToolchain.From(dotnetCli64bit)).WithId("64 bit cli"));
        }
    }

    [Params(false, true)]
    public bool CallStopwatchTimestamp { get; set; }

    [GlobalSetup]
    public void GlobalSetup()
    {
        if (CallStopwatchTimestamp)
            Stopwatch.GetTimestamp();
    }

    private const int IterationCount = 10001;

    [Benchmark(OperationsPerInvoke = IterationCount)]
    public string WithStopwatch()
    {
        double a = 1, b = 1;
        var sw = new Stopwatch();
        for (int i = 0; i < IterationCount; i++)
        {
            // fld1
            // fadd        qword ptr [ebp-0Ch]
            // fstp        qword ptr [ebp-0Ch]
            a = a + b;
        }
        return string.Format("{0}{1}", a, sw.ElapsedMilliseconds);
    }

    [Benchmark(OperationsPerInvoke = IterationCount)]
    public string WithoutStopwatch()
    {
        double a = 1, b = 1;
        for (int i = 0; i < IterationCount; i++)
        {
            // fld1
            // faddp       st(1),st
            a = a + b;
        }
        return string.Format("{0}", a);
    }
}

Sample results:

BenchmarkDotNet=v0.10.9.20170910-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742185 Hz, Resolution=364.6727 ns, Timer=TSC
.NET Core SDK=2.1.0-preview1-007074
  [Host]     : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
  32 bit cli : .NET Core 2.0.0 (Framework 4.6.00001.0), 32bit RyuJIT
  64 bit cli : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT

Jit=RyuJit  

Method	Job	Platform	Toolchain	CallStopwatchTimestamp	Mean	Error	StdDev
WithStopwatch	32 bit cli	X86	32 bit cli	False	2.862 ns	0.0552 ns	0.0657 ns
WithoutStopwatch	32 bit cli	X86	32 bit cli	False	2.771 ns	0.0435 ns	0.0407 ns
WithStopwatch	64 bit cli	X64	64 bit cli	False	1.314 ns	0.0194 ns	0.0172 ns
WithoutStopwatch	64 bit cli	X64	64 bit cli	False	1.256 ns	0.0187 ns	0.0175 ns
WithStopwatch	32 bit cli	X86	32 bit cli	True	2.841 ns	0.0568 ns	0.0995 ns
WithoutStopwatch	32 bit cli	X86	32 bit cli	True	2.765 ns	0.0361 ns	0.0338 ns
WithStopwatch	64 bit cli	X64	64 bit cli	True	1.298 ns	0.0256 ns	0.0314 ns
WithoutStopwatch	64 bit cli	X64	64 bit cli	True	1.309 ns	0.0179 ns	0.0168 ns

[AndreyAkinshin]

Great, thanks! However, discovering still can be a great feature, it will allow making benchmarks more portable. Maybe we can create another up-for-grabs issue?

[adamsitnik]

@AndreyAkinshin it would require handling Windows/Mac/Linux and custom paths. What if user has few dotnet cli installed? Which should we choose then?

It's possible to handle all of these situations but so far I was the only person that wanted to use it, the chance for having some contributor willing to implement it (and do it correctly) is very low to me.

On the other hand, now we can allow people to experiment with different versions of dotnet cli.