BenchmarkDotNet cannot run tests on latest SDK

dotnet / runtimelab

This repo is for experimentation and exploring new ideas that may or may not make it into the main dotnet/runtime repo.

MIT License

1.42k stars 198 forks source link

BenchmarkDotNet cannot run tests on latest SDK #1236

Closed kant2002 closed 3 years ago

kant2002 commented 3 years ago

Followup on https://github.com/dotnet/BenchmarkDotNet/pull/1724

If I create minimal benchmark and compile it on following SDKs

6.0.100-preview.6.21304.1
5.0.300

5.0.301 then benchmark project fails with

error NETSDK1150: The referenced project 'CoreRtBenchmark50300.csproj' is a non self-contained executable.  A non self-contained executable cannot be referenced by a self-contained executable.

If I run on 5.0.204 SDK then benchmarks run successfully.

kant2002 commented 3 years ago

Based on @MichalStrehovsky remark that means that NativeAOT probably should ignore that kind of errors. Does that means since NativeAOT produce single file then all references does not matter?

MichalStrehovsky commented 3 years ago

@adamsitnik This kind of looks like Benchmark.NET is going to fail once the CI machine updates to the latest .NET 5 SDK? Maybe we need to reopen https://github.com/dotnet/BenchmarkDotNet/pull/1724 after all? The error message makes it look like a problem that is unique to how Benchmark.NET is setup (there is a project that references the CoreRT project?)

kant2002 commented 3 years ago

According to logs, currently integration test for CoreRT on BenchmarkDotNet repo use 5.0.101 SDK just in case if this would be handy.

JoshuaWierenga commented 3 years ago

I had this error a week ago, https://docs.microsoft.com/en-us/dotnet/core/compatibility/sdk/5.0/referencing-executable-generates-error appears a reasonable solution in the short term. I have only seen this error in places where it doesn't belong since the comment about the build not being executable always ends up being wrong. I can at least confirm that it isn't specific to BenchmarkDotNet though.

adamsitnik commented 3 years ago

@kant2002 could you please reopen dotnet/BenchmarkDotNet#1724 and update the SDK version used by BDN CI?

https://github.com/dotnet/BenchmarkDotNet/blob/5196e6f5854c45b3e62a8c9a0a68e92491e70d57/build.ps1#L35 https://github.com/dotnet/BenchmarkDotNet/blob/5196e6f5854c45b3e62a8c9a0a68e92491e70d57/build.sh#L50