codewars / runner

Issue tracker for Code Runner
34 stars 8 forks source link

PowerShell tests fail with no feedback #290

Open hobovsky opened 10 months ago

hobovsky commented 10 months ago

Some kinds of errors cause PowerShell tests fail with completely no feedback other than "Time: 2277ms Response received but no data was written to STDOUT or STDERR.".

Example: this kumite.

I think this happens mostly on syntax errors, but I am not sure if it's the only kind of errors causing this, is it limited only to some snippets, etc.

kaz-yoshihara commented 10 months ago

We're converting the NUnitXML, so that's probably why we get nothing if the test doesn't start. We can change how we run the tests if there's a way to capture the errors.


Here's how we run the tests.

The files:

The command:

pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass Run-Tests.ps1

Run-Tests.ps1:

Import-Module -Name Pester
Import-Module -Name Pester-Codewars

$config = New-PesterConfiguration
$config.Should.ErrorAction = 'Continue'
$config.Output.Verbosity = 'None'
$config.Run.PassThru = $true

Invoke-Pester -Configuration $config | ConvertTo-NUnitReport | ConvertTo-Codewars

Pester-Codewars is https://github.com/codewars/Pester-Codewars and stored in ~/.local/share/powershell/Modules/Pester-Codewars/Pester-Codewars.psm1.

hobovsky commented 10 months ago

After the Invoke-Pester finishes, something takes care of processing the NUnit XML, right? Is it possible to check out the stdout/stderr if the report is not found, and present it as output of a test run?

I can check if/how such error is reported, I just do not know if it's possible to use stdout if there's no XML report.

EDIT: Okay I think I got it.

kazk commented 10 months ago

The last line in Run-Tests.ps1:

Invoke-Pester -Configuration $config | ConvertTo-NUnitReport | ConvertTo-Codewars

I forgot the details, but ConvertTo-NUnitReport should be generating the NUnit XML. Then that's piped to ConvertTo-Codewars . I don't think we do anything to the stderr.