Benchmark validation doesn't appear to be working on non-Windows OS

[JamesNK]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

Benchmarks written with benchmarkdotnet are validated on build:

https://github.com/dotnet/aspnetcore/blob/f0af11263e006d0df75b32d20d85bde5870a8d9c/eng/targets/CSharp.Common.targets#L133-L142

It appears that failures are only correctly reported on Windows. This behavior was noticed when I added a benchmark that is only valid for Windows, but tests all passed.

For example, I added an exception to a benchmark's setup here - https://github.com/dotnet/aspnetcore/pull/46491 - and only Windows tests failed:

I can see in the logs that they are being detected and run on other OS, but no error reported:

Expected Behavior

Validation reports problems and the build fails on Linux and macOS.

Steps To Reproduce

https://github.com/dotnet/aspnetcore/pull/46491

Exceptions (if any)

No response

.NET Version

No response

Anything else?

No response

[JamesNK]

cc @BrennanConroy

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.