GD-138: Add capture test case execution stdout to the test report

[MikeSchulze]

Why

https://github.com/MikeSchulze/gdUnit4Net/issues/138

What

• introduce a new test setting option CaptureStdOut
• added a stdout/console hook to capture Console messages and Godot Print messages
• extend the test report by add new type STDOUT to provide collected stdout messages
• do handle reports more IDE specific to get better report results

[van800]

I have tried this branch. Logging Console.WriteLine seem to be working only in Windows.

[MikeSchulze]

I have tried this branch. Logging Console.WriteLine seem to be working only in Windows.

It's not done yet, I'm actually working on the Unix stdout capture

[MikeSchulze]

I have actual big issues to debug on Mac with rider, it often does not close the debug subprocess and need to close it manually. Can you confirm the same issue on your test env?

[van800]

I don't see that behavior of not closed debug subprocess.

[van800]

It reproduced now without debugging.

[van800]

Stopped by itself after some 10-30 seconds.

[MikeSchulze]

Stopped by itself after some 10-30 seconds.

@van800 There are different issues.

At first, the explorer was not responding after test run for 5s I had a timeout of 5s on the report server wait, I set it now to 1s so it finishes direct after execution. I need to investigate why the report server is so long in wait state. There is an exception in the log

16:15:26.256 |I| TestRunner: JetBrains.ReSharper.TestRunner.Adapters.VsTest.VsTestRunner Execution completed 
16:15:26.262 |E| TestRunner: JetBrains.ReSharper.TestRunner.Adapters.VsTest.Executor GdUnit4.TestEventReportServer:: StackTrace:    at System.IO.Pipes.PipeStream.CheckReadOperations()
   at System.IO.Pipes.PipeStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)
   at GdUnit4.TestAdapter.Execution.TestEventReportServer.Start(IFrameworkHandle frameworkHandle, IReadOnlyList`1 tests) in /Users/mikeschulze/Documents/develop/workspace/gdUnit4Net/testadapter/src/execution/TestEventReportServer.cs:line 64 
16:15:26.283 |I| TestRunner: JetBrains.ReSharper.TestRunner.Adapters.VsTest.Executor GdUnit4.TestEventReportServer:: Disconnected. 

And the debugger is not quit until the configured session timeout (default 30s) At WaitForExit it waits, but the underlying process was already terminated

  private void RunDebugRider(IFrameworkHandle2 fh2, ProcessStartInfo psi)
    {
        // EnableShutdownAfterTestRun is not working we need to use SafeHandle the get the process running until the ExitCode is getted
        fh2.EnableShutdownAfterTestRun = true;
        fh2.SendMessage(TestMessageLevel.Informational, $"Debug process started {psi.FileName} {psi.WorkingDirectory} {psi.Arguments}");
        var processId = fh2.LaunchProcessWithDebuggerAttached(psi.FileName, psi.WorkingDirectory, psi.Arguments, psi.Environment);
        pProcess = Process.GetProcessById(processId);
        SafeProcessHandle? processHandle = null;
        try
        {
            processHandle = pProcess.SafeHandle;
            var isExited = pProcess.WaitForExit(SessionTimeOut);
            // it never exits on macOS ?
            //fh2.SendMessage(TestMessageLevel.Informational, $"Process exited: HasExited: {pProcess.HasExited} {isExited} {processHandle}");
            // enforce kill the process has also no affect on macOS
            pProcess.Kill(true);
            //fh2.SendMessage(TestMessageLevel.Informational, $"Process exited: HasExited: {pProcess.HasExited} {processHandle.IsClosed}");
            // this line fails on macOS, maybe the SafeHandle works only on windows
            //fh2.SendMessage(TestMessageLevel.Informational, @$"Run TestRunner ends with {pProcess.ExitCode}");
        }
        catch (Exception e)
        {
            fh2.SendMessage(TestMessageLevel.Error, e.Message);
        }
        finally
        {
            processHandle?.Dispose();
        }
    }

[van800]

https://chatgpt.com/share/67190b4c-2ec0-800c-912b-1c323a6a5c72 please try the suggested workarounds. It says HasExited is reliable on Mac and WaitForExit is not.

[MikeSchulze]

https://chatgpt.com/share/67190b4c-2ec0-800c-912b-1c323a6a5c72 please try the suggested workarounds. It says HasExited is reliable on Mac and WaitForExit is not.

I had tried already to use pProcess.HasExited but without success

[van800]

Turns out I only see the delay on this line testEventServerTask.Wait(TimeSpan.FromSeconds(5)); changing it to 0.1 seconds seems to be working fine.

Only place, where I was confused for a bit was when debugging both test and the framework, there are 2 debugging sessions. When the test finishes, focus moves to the test-runner, while there is a second active debug tab. At the sceenshot the other tab is running and waiting at a breakpoint.

[MikeSchulze]

Turns out I only see the delay on this line testEventServerTask.Wait(TimeSpan.FromSeconds(5)); changing it to 0.1 seconds seems to be working fine.

Only place, where I was confused for a bit was when debugging both test and the framework, there are 2 debugging sessions. When the test finishes, focus moves to the test-runner, while there is a second active debug tab. At the sceenshot the other tab is running and waiting at a breakpoint.

I will open a separate pull request to fix the debug/timing issues.