search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.61k stars 973 forks source link

Starting CLI Commands no longer works via the dotnet-runtime-host ".exe" #2664

Open matthid opened 2 weeks ago

matthid commented 2 weeks ago

Since 0,14 and the new build system it seems like you can no longer start other referenced cli projects via Process.Start in BenchmarkDotNet.

Background: It seems like the embedded dll is always the one from BenchmarkDotNet now.

This is a COREHOST_TRACE=1 capture of the started project

Tracing enabled @ Wed Nov 13 14:41:27 2024 GMT
--- Invoked apphost [version: 8.0.11 @Commit: 9cb3b725e3ad2b57ddc9fb2dd48d2d170563a8f5] main = {
C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\bin\Release\net8.0-Windows7.0\OtherProject.exe
config=C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\bin\Release\net8.0-Windows7.0\epsSingle.config
}
Redirecting errors to custom writer.
The managed DLL bound to this executable is: 'f4d0ff40-a7b1-4376-b14d-87ef1f70a014.dll'

The problem is it should be 'OtherProject.dll' instead since that is a completely other project.

The problem only materializes when you use a command line which can not be parsed by the BDN dll and your output will be:

// BeforeAnythingElse

System.FormatException: The input string 'config=C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\bin\Release\net8.0-Windows7.0\epsSingle.config' was not in a correct format.
   at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value)
   at System.Int32.Parse(String s)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Dev\TestProj\Src\TestPerfRunner\bin\Net6\Release\net8.0-windows\f4d0ff40-a7b1-4376-b14d-87ef1f70a014\f4d0ff40-a7b1-4376-b14d-87ef1f70a014.notcs:line 45
// AfterAll

Instead of running the correct entry point.

Workaround:

we now start dotnet <OtherProject.dll> instead but it was quite the pain to find and debug this problem, since if you open the .exe or the .dll in DotPeek everything looks nice, and if you don't have actual parameters it will forward to the 'correct' entrypoint.

timcassell commented 2 weeks ago

@matthid I don't really follow your usage explanation. Could you share a repro project?

matthid commented 1 week ago

Sure: https://github.com/matthid/BNDIssue2664 here is a repro to reproduce the issue. Like I said it basically is just Process.Start with the .exe of an unrelated project.

timcassell commented 1 week ago

I see the exe in the binary output directory as expected. However, when I run it directly, it prints:

Details

```cmd // Benchmark Process Environment Information: // BenchmarkDotNet v0.13.13-develop (2024-11-22) // Runtime=.NET 8.0.10 (8.0.1024.46610), X64 RyuJIT SSE3 // GC=Concurrent Workstation // HardwareIntrinsics=SSE3,LZCNT VectorSize=128 // Job: DefaultJob System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: External process failed (-1): windir: C:\WINDOWS LOCALAPPDATA: C:\Users\Tim\AppData\Local CommonProgramFiles: C:\Program Files\Common Files PROCESSOR_REVISION: 0a00 FP_NO_HOST_CHECK: NO OS: Windows_NT PROCESSOR_ARCHITECTURE: AMD64 SystemDrive: C: TMP: C:\Users\Tim\AppData\Local\Temp LOGONSERVER: \\TIM-PC OneDrive: C:\Users\Tim\OneDrive USERDOMAIN: Tim-PC ProgramFiles: C:\Program Files ProgramW6432: C:\Program Files HOMEPATH: \Users\Tim USERDOMAIN_ROAMINGPROFILE: Tim-PC DriverData: C:\Windows\System32\Drivers\DriverData ProgramData: C:\ProgramData SESSIONNAME: Console CommonProgramW6432: C:\Program Files\Common Files USERPROFILE: C:\Users\Tim PROCESSOR_LEVEL: 16 ADSK_3DSMAX_x64_2018: C:\Program Files\Autodesk\3ds Max 2018\ GIT_LFS_PATH: C:\Program Files\Git LFS PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL PUBLIC: C:\Users\Public CommonProgramFiles(x86): C:\Program Files (x86)\Common Files TEMP: C:\Users\Tim\AppData\Local\Temp ComSpec: C:\WINDOWS\system32\cmd.exe SystemRoot: C:\WINDOWS DEPOT_TOOLS_WIN_TOOLCHAIN: 0 ProgramFiles(x86): C:\Program Files (x86) OneDriveConsumer: C:\Users\Tim\OneDrive ChocolateyLastPathUpdate: 133254941067942229 USERNAME: Tim COMPUTERNAME: TIM-PC APPDATA: C:\Users\Tim\AppData\Roaming PROCESSOR_IDENTIFIER: AMD64 Family 16 Model 10 Stepping 0, AuthenticAMD ALLUSERSPROFILE: C:\ProgramData HOMEDRIVE: C: NUMBER_OF_PROCESSORS: 6 ChocolateyInstall: C:\ProgramData\chocolatey PSModulePath: C:\Users\Tim\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\ GTK_BASEPATH: C:\Program Files (x86)\GtkSharp\2.12\ Path: C:\Users\Tim\source\depot_tools;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Program Files\Common Files\Autodesk Shared\;C:\Program Files (x86)\Autodesk\Backburner\;C:\Program Files\Git LFS;C:\Program Files\Mono\bin;C:\Program Files (x86)\GtkSharp\2.12\bin;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Users\Tim\.dotnet\tools;C:\Program Files (x86)\NUnit.org\nunit-console;C:\Python27\Scripts;C:\Program Files\Perforce\;C:\Program Files (x86)\dotnet-core-uninstall\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;C:\Program Files\CMake\bin;C:\Users\Tim\.jsvu\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Mono\bin;C:\Users\Tim\.dotnet\tools;C:\Program Files (x86)\NUnit.org\nunit-console;C:\Users\Tim\AppData\Local\Microsoft\WindowsApps;C:\Users\Tim\AppData\Roaming\npm;C:\Users\Tim\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Tim\.dotnet\tools;C:\Users\Tim\.dotnet\tools;C:\Users\Tim\.dotnet\tools;C:\Users\Tim\.jsvu\bin;C:\Users\Tim\.dotnet\tools OUT:// BeforeAnythingElse OUT: OUT:System.FormatException: The input string 'config=test.config' was not in a correct format. OUT: at System.Number.ThrowFormatException[TChar](ReadOnlySpan`1 value) OUT: at System.Int32.Parse(String s) OUT: at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\bin\Release\net8.0\d2dbb57d-d694-4e59-9f73-9f7e5480c263\d2dbb57d-d694-4e59-9f73-9f7e5480c263.notcs:line 45 OUT:// AfterAll at Benchmark.TestBenchmarkWithExternalProc.GlobalSetup() in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\TestBenchmarkWithExternalProc.cs:line 57 at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters) in C:\BenchmarkDotNet\src\BenchmarkDotNet\Engines\EngineFactory.cs:line 28 at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName) in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\bin\Release\net8.0\d2dbb57d-d694-4e59-9f73-9f7e5480c263\d2dbb57d-d694-4e59-9f73-9f7e5480c263.notcs:line 176 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr) --- End of inner exception stack trace --- at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr) at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\Tim\Downloads\BNDIssue2664-main\Benchmark\bin\Release\net8.0\d2dbb57d-d694-4e59-9f73-9f7e5480c263\d2dbb57d-d694-4e59-9f73-9f7e5480c263.notcs:line 57 // AfterAll ```

It looks like it's being overwritten with the benchmark exe. So it's trying to process spawn itself recursively. And since it's actually the benchmark code running instead of the expected process, it doesn't recognize the arguments and throws.

@rainersigwald Any idea why passing the output path build arguments would cause this?

matthid commented 1 week ago

Just to add something which might be a hint: Without any parameter it actually starts the correct program (you can add a Console.WriteLine to see the correct output). So the generated code later starts the correct dll, so one workaround/hack might be to change what BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached does.

But indeed the correct way would be to have it emit a valid exe host with the correct dll binding (as you can see in my trace output above)