CoreCLR Compatibility

[smudge202]

From what I've read, there has been some work regarding CoreCLR support. I don't know the full details but I'm pretty sure that dependencies on the following assemblies will present a problem:

• System.Management
• Microsoft.Diagnostic.Runtime
• Dia2Lib
• Interop.Dia2Lib

As per the discussion in #33, I believe @mattwarren will be moving those dependencies. I'm assuming this will allow us to work towards a CoreCLR compatible implementation which can be interchanged in for the consumers of the above libraries?

Note: This does not relate to DNX Compatibility directly. Supporting DNX would make it easier to implement CoreCLR compatibility purely because the tooling is so much better for cross compilation, but probably isn't a showstopper.

[mattwarren]

As per the discussion in #33, I believe @mattwarren will be moving those dependencies. I'm assuming this will allow us to work towards a CoreCLR compatible implementation which can be interchanged in for the consumers of the above libraries?

Yeah that's the intention in the long-run. Basically the BenchmarkDotNet.Diagnostic dll will be loaded dynamically, but only on run-times that it support.

Does CoreCLR/DNX support dynamically loading dll/assembelies?

[smudge202]

Define dynamically loading assemblies (by what method)? You can for example create assemblies and types in-memory as shown here (poorly): https://github.com/smudge202/compose/blob/dev/src/Compose/DynamicEmitter.cs

[mattwarren]

Sorry, "dynamically" was a bad choice of words.

I didn't mean dynamically creating/loading types in-memory, what I meant was something like this SO answer:

Assembly assembly = Assembly.LoadFrom("MyNice.dll");
Type type = assembly.GetType("MyType");
object instanceOfMyType = Activator.CreateInstance(type);
ISomeInterface instanceOfInterface = (ISomeInterface)instanceOfMyType.Unwrap()

[smudge202]

Yup. To my knowledge you can defer assembly loading to runtime as shown just fine on CoreCLR. I can throw together a 2 minute example tonight to confirm.

[smudge202]

Just to update following stealing a lot of @mattwarren 's time this evening in gitter, this will be very difficult to achieve due to indirect dependencies on Win32 (Microsoft.Diagnostics.Runtime \ ClrMD P/Invokes into Win32). CoreCLR is intended to run cross platform, so I wouldn't expect those calls to become available on CoreCLR in the foreseeable future, if ever.

[mattwarren]

@smudge202 @AndreyAkinshin

I've just checked in the changes the de-couple BenchmarkDotNet.Diagnostics from code, so all the problematic dependencies have gone away. You should be free to do the CoreCLR/DNX stuff now, any problems let me know.

[AndreyAkinshin]

@mattwarren, great!

[damageboy]

+1

Loving to try this out on a project I'm converting now

[adamsitnik]

@AndreyAkinshin You can assign me for this issue. I just got it running on Windows ;)

[adamsitnik]

I have implemented .NET Core support for BDN based on existing Dnx toolchain. Code is here

The only limitation I have encountered is that dotnet cli supports only x64 compilation as for now. So currently I ignore the specified processor architecture and always set x64.

I know that this is wrong. What do you think I should do with 32 bit Benchmarks? Filter them out and notify user? Or compile anyway and try to execute? Then in future we do not need to adjust BDN to dotnet cli changes.

I have tested it only for Windows as for now.

[AndreyAkinshin]

@adamsitnik, it is awesome, thank you again.

However, I have some questions/suggestions.

1. Is it possible to create a benchmark with two Jobs: Dnx and Core? I really want to run it as a single benchmark.
2. Maybe we can merge DnxToolchain with CoreToolchain, and use BenchmarkDotNet.Jobs.Runtime?
3. For now, all exporters create files (md, csv, png) in the project directory (I see them in the Solution Explorer). It would be cool to keep these files in a separated folder.
4. Can we determine version of CoreCLR and print it in the environment info?

---

I tested in on Windows, and I already have some interesting results about CoreCLR:

BenchmarkDotNet-Dev=v0.9.2.0+
OS=Microsoft Windows NT 10.0.10586.0
Processor=Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz, ProcessorCount=8
Frequency=2728061 ticks, Resolution=366.5607 ns
HostCLR=MS.NET 4.0.30319.42000, Arch=32-bit RELEASE

Type=Algo_Md5VsSha256  Mode=Throughput  Toolchain=Dnx  

Method	Median	StdDev
Md5	23.5390 us	1.3793 us
Sha256	131.2897 us	8.7399 us

BenchmarkDotNet-Dev=v0.9.2.0+
OS=Windows
Processor=?, ProcessorCount=8
Frequency=2728061 ticks, Resolution=366.5607 ns
HostCLR=CORE, Arch=32-bit RELEASE

Type=Algo_Md5VsSha256  Mode=Throughput  Toolchain=Core  

Method	Median	StdDev
Md5	19.6306 us	0.3011 us
Sha256	49.2485 us	1.2627 us

---

For now, I have some troubles with CoreCLR on Linux. It seems that dotnet cli doesn't work well on modern version of Ubuntu (see Ubuntu 15.10 can't install dotnet). I will try to run cli+BDN on Ubuntu 14.04 soon.

[adamsitnik]

1. Is it possible to create a benchmark with two Jobs: Dnx and Core? I really want to run it as a single benchmark.
2. Maybe we can merge DnxToolchain with CoreToolchain, and use BenchmarkDotNet.Jobs.Runtime?

Good idea! I will implement it.

For now, all exporters create files (md, csv, png) in the project directory (I see them in the Solution Explorer). It would be cool to keep these files in a separated folder.

4. Can we determine version of CoreCLR and print it in the environment info?

I will check this out.

I tested in on Windows, and I already have some interesting results about CoreCLR:

I believe it could be because of the different processor architecture. For Core we always compile to x64, so here it could be 32bit dnx vs 64 bit core problem. I will push some updates for this soon

I will try to run cli+BDN on Ubuntu 14.04 soon.

This is great!

[adamsitnik]

@AndreyAkinshin I have updated coreclr branch.

Changes:

• Core - not supported benchmarks are not executed (32bit, LegacyJit)
• Core and Dnx as Runtimes, not Toolchains
• I removed Toolchain from configuraiton, but the internal implementation is still using Toolchains, which are used based on Runtime

Sample config file:

private class DnxAndCoreConfig : ManualConfig
{
    public DnxAndCoreConfig()
    {
        Add(Job.Dry.With(Runtime.Dnx).With(Jit.Host));
        Add(Job.Dry.With(Runtime.Core).With(Jit.Host));
    }
}

Sample results:

    Method | Runtime |     Median |    StdDev |
---------- |-------- |----------- |---------- |
 Benchmark |    Core | 12.4020 us | 0.0000 us |
 Benchmark |     Dnx | 13.2573 us | 0.0000 us |

[AndreyAkinshin]

Awesome!

[AndreyAkinshin]

It looks like everything works fine. And results for my favorite benchmark:

BenchmarkDotNet-Dev=v0.9.2.0+
OS=Microsoft Windows NT 10.0.10586.0
Processor=Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz, ProcessorCount=8
Frequency=14318180 ticks, Resolution=69.8413 ns
HostCLR=DNX MS.NET 4.0.30319.42000, Arch=32-bit RELEASE

Type=Algo_Md5VsSha256  Mode=Throughput  

Method	Runtime	Median	StdDev
Md5	Core	24.5509 us	1.1201 us
Md5	Dnx	26.5151 us	1.0375 us
Sha256	Core	58.1015 us	1.5358 us
Sha256	Dnx	132.4039 us	3.4255 us

Are we ready to merge it into master?

[redknightlois]

Dnx is running DesktopCLR?

[AndreyAkinshin]

@adamsitnik, I tried to run BDN on Ubuntu 14.04. Good news: master process works! Bad news: we have some troubles with Generator (however, it looks like a minor issue). On Linux, BDN create directory ..\ for generated benchmarks, see attach.

[adamsitnik]

@AndreyAkinshin Good news! It is great that you can test it.

Could you try to update DotNetCliGenerator in following way:

protected override string GetDirectoryPath(Benchmark benchmark)
        {
            return Path.Combine(
                new DirectoryInfo(Directory.GetCurrentDirectory()).Parent.FullName, 
                benchmark.ShortInfo);
        }

[adamsitnik]

Dnx is running DesktopCLR?

@redknightlois Yes, exactly. DNX451 is a DNX SDK running on .Net 4.5.1 (Desktop CLR / Full BCL and FCL)

[AndreyAkinshin]

@adamsitnik, success! CoreCLR benchmark runs on Linux. There is only one issue: it seems that without root privileges we can't change priority of the process. I can run benchmark only with help of sudo.

[adamsitnik]

@AndreyAkinshin HURRAY! ;D Do you have any idea how to overcome the priority problem?

[AndreyAkinshin]

My suggestion for today: can we just wrap it in try-catch?

[adamsitnik]

Ok. I will add try catch and log warning on exception with the sudo hint.

[adamsitnik]

Btw I wonder if setting the affinity would also throw

[AndreyAkinshin]

Probably. My plan for nearest future:

1. Publish a library version with CoreCLR support on NuGet
2. Merge coreclr into develop, and publish a version with last develop changes.
3. Fix bugs one by one.

[adamsitnik]

@AndreyAkinshin I believe that it is a good plan. You can count on me with merge and bug fixing.

I created PR to have some more clear track of what changes have been introduced.