codewars / runner

Issue tracker for Code Runner
32 stars 8 forks source link

PowerShell tests fail with no feedback #290

Open hobovsky opened 5 months ago

hobovsky commented 5 months ago

Some kinds of errors cause PowerShell tests fail with completely no feedback other than "Time: 2277ms Response received but no data was written to STDOUT or STDERR.".

Example: this kumite.

I think this happens mostly on syntax errors, but I am not sure if it's the only kind of errors causing this, is it limited only to some snippets, etc.

kaz-yoshihara commented 5 months ago

We're converting the NUnitXML, so that's probably why we get nothing if the test doesn't start. We can change how we run the tests if there's a way to capture the errors.


Here's how we run the tests.

The files:

The command:

pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass Run-Tests.ps1

Run-Tests.ps1:

Import-Module -Name Pester
Import-Module -Name Pester-Codewars

$config = New-PesterConfiguration
$config.Should.ErrorAction = 'Continue'
$config.Output.Verbosity = 'None'
$config.Run.PassThru = $true

Invoke-Pester -Configuration $config | ConvertTo-NUnitReport | ConvertTo-Codewars

Pester-Codewars is https://github.com/codewars/Pester-Codewars and stored in ~/.local/share/powershell/Modules/Pester-Codewars/Pester-Codewars.psm1.

hobovsky commented 5 months ago

After the Invoke-Pester finishes, something takes care of processing the NUnit XML, right? Is it possible to check out the stdout/stderr if the report is not found, and present it as output of a test run?

I can check if/how such error is reported, I just do not know if it's possible to use stdout if there's no XML report.

EDIT: Okay I think I got it.

kazk commented 5 months ago

The last line in Run-Tests.ps1:

Invoke-Pester -Configuration $config | ConvertTo-NUnitReport | ConvertTo-Codewars

I forgot the details, but ConvertTo-NUnitReport should be generating the NUnit XML. Then that's piped to ConvertTo-Codewars . I don't think we do anything to the stderr.