pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.08k stars 470 forks source link

Error during handling of output-element within Write-NUnit3OutputElement #2421

Closed akaer closed 4 months ago

akaer commented 8 months ago

Checklist

What is the issue?

Depending on the type of output the processing might fail during calling the ToString() method within Write-NUnit3OutputElement. As soon as this error occurs, the test result file (TestResult.OutputPath) is no longer written completely.

function Write-NUnit3OutputElement ($Output, [System.Xml.XmlWriter] $XmlWriter) {
    $outputString = @(foreach ($o in $Output) { $o.ToString() }) -join [System.Environment]::NewLine

    $XmlWriter.WriteStartElement('output')
    $XmlWriter.WriteCData($outputString)
    $XmlWriter.WriteEndElement()
}

Stacktrace might be

System.Management.Automation.RuntimeException: Invoking step End failed:
Result 1 - Error 1:Export-XmlReport : You cannot call a method on a null-valued expression.
At C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1:17627 char:13
+             Export-XmlReport -Result $Result -Path $Path -Format $For ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Export-XmlReport], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull,Export-XmlReport

at Write-NUnit3OutputElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17490
at Write-NUnit3TestCaseElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17438
at Write-NUnit3TestSuiteElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17142
at Write-NUnit3TestSuiteElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17099
at Write-NUnit3TestRunChildNode, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17000
at Write-NUnit3Report, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 16953
at Export-XmlReport, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17779
at Export-PesterResult, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17627
at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 18120
at Invoke-PluginStep, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1788
at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 5077
at <ScriptBlock>, C:\temp\Pester_Nunit3_Bug\RunPester.ps1: line 10
at <ScriptBlock>, <No file>: line 1

at Assert-Success, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1845
at Invoke-PluginStep, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1807
at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 5077
at <ScriptBlock>, C:\temp\Pester_Nunit3_Bug\RunPester.ps1: line 10
at <ScriptBlock>, <No file>: line 1

or another example

System.Management.Automation.RuntimeException: Invoking step End failed:
Result 1 - Error 1:Export-XmlReport : Exception calling "ToString" with "0" argument(s): "An event was unable to invoke any of the subscribers (Exception from HRESULT: 0x80040201)"
At C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1:17627 char:13
+             Export-XmlReport -Result $Result -Path $Path -Format $For ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Export-XmlReport], MethodInvocationException
    + FullyQualifiedErrorId : COMException,Export-XmlReport

at Write-NUnit3OutputElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17490
at Write-NUnit3TestCaseElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17438
at Write-NUnit3TestSuiteElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17142
at Write-NUnit3TestSuiteElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17099
at Write-NUnit3TestSuiteElement, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17099
at Write-NUnit3TestRunChildNode, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17000
at Write-NUnit3Report, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 16953
at Export-XmlReport, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17779
at Export-PesterResult, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 17627
at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 18120
at Invoke-PluginStep, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1788
at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 5077
at <ScriptBlock>, C:\projects\OneIM\v90\TST\Tests\End2End\E2E.DataImporter.Posh\RunPester.ps1: line 26
at <ScriptBlock>, <No file>: line 1

at Assert-Success, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1845
at Invoke-PluginStep, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1807
at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 5077
at <ScriptBlock>, C:\projects\OneIM\v90\TST\Tests\End2End\E2E.DataImporter.Posh\RunPester.ps1: line 26
at <ScriptBlock>, <No file>: line 1

Expected Behavior

Complete test result file (TestResult.OutputPath) is written. If there might be errors during processing the output-element might be skipped.

Steps To Reproduce

$sb = {
    Describe 'bug' {

        Context 'Success' {
            It 'Can bar' {
                'Hello World 1'
                'Hello World 2'
                1 | Should -Be 1
            }
        }

        Context 'Failure' {
            It 'Can foo' {
                'Hello World 1'
                $null
                'Hello World 2'
                1 | Should -Be 1
            }
        }

        Context 'No more results written into output file after failure' {
            It 'Can baz' {
                'Hello World 1'
                'Hello World 2'
                1 | Should -Be 1
            }
        }

    }
}

$container = New-PesterContainer -ScriptBlock $sb

$config = New-PesterConfiguration
$config.CodeCoverage.Enabled = $false
$config.TestResult.Enabled = $true
$config.TestResult.OutputFormat = 'NUnit3'
$config.TestResult.OutputPath = 'TestResults.xml'
$config.TestResult.TestSuiteName = 'FooBar'
$config.Output.Verbosity = 'Detailed'
$config.Run.Container = $container

Invoke-Pester -Configuration $config

Describe your environment

Pester version : 5.5.0 C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1 PowerShell version : 5.1.22621.2506 OS version : Microsoft Windows NT 10.0.22631.0

Possible Solution?

Only a suggestion. Maybe check for not null in combination with try / silent catch like

function Write-NUnit3OutputElement ($Output, [System.Xml.XmlWriter] $XmlWriter) {
    $outputString = @(foreach ($o in $Output) { 
    if ($o) {
        try {
            $o.ToString()
        }
        catch {
        }
    }
    }) -join [System.Environment]::NewLine

    $XmlWriter.WriteStartElement('output')
    $XmlWriter.WriteCData($outputString)
    $XmlWriter.WriteEndElement()
}
fflaten commented 8 months ago

Thanks for the detailed report. Will get this fixed in next release.

Suggestion looks good. I'm wondering if $o -as [string] might work as well. Need to double check behavior differences besides ignoring null and compatibility.