search

pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

Code coverage fails in some circumstances when UseBreakpoints is `$false` #2306

Open johlju opened 1 year ago

johlju commented 1 year ago

Checklist

What is the issue?

I started seening this issue when I started doing class-based resource ~1 year ago, and since it has been troublesome to make a repro case so I just switched to using breakpoints for code coverage instead. But now I managed to repro it with simpler code and simpler tests.

This test works when CodeCoverage.UseBreakpoints = $true but fails when CodeCoverage.UseBreakpoints = $false.

There seems to be a problem somewhere inside Pester when using this new code coverage method.

Expected Behavior

The tests to pass regardless if UseBreakpoints is set to $true or $false.

Steps To Reproduce

File SqlServerDsc.psm1

class StartupParameters
{
    [System.UInt32[]]
    $TraceFlag

    static [StartupParameters] Parse([System.String] $InstanceStartupParameters)
    {
        $startupParameters = [StartupParameters]::new()

        $startupParameterValues = $InstanceStartupParameters -split ';'

        $startupParameters.TraceFlag = [System.UInt32[]] @(
            $startupParameterValues |
                Where-Object -FilterScript {
                    $_ -match '^-T\d+'
                } |
                ForEach-Object -Process {
                    [System.UInt32] $_.TrimStart('-T')
                }
        )

        return $startupParameters
    }
}

function Get-SqlDscTraceFlag
{
    [OutputType([System.UInt32[]])]
    [CmdletBinding(DefaultParameterSetName = 'ByServerName')]
    param
    (
        [Parameter(ParameterSetName = 'ByServiceObject', Mandatory = $true)]
        [Object]
        $ServiceObject
    )

    $traceFlags = [System.UInt32[]] @()

    if ($ServiceObject.StartupParameters)
    {
        $traceFlags = [StartupParameters]::Parse($ServiceObject.StartupParameters).TraceFlag
    }

    return , [System.UInt32[]] $traceFlags
}

File SqlServerDsc.Tests.ps1

BeforeAll {
    Import-Module -Name './SqlServerDsc.psm1' -Force
}

AfterAll {
    # Unload the module being tested so that it doesn't impact any other tests.
    Get-Module -Name 'SqlServerDsc' -All | Remove-Module -Force
}

Describe 'SqlServerDsc' {
    Context 'When one trace flag exist' {
        BeforeAll {
            $mockStartupParameters = '-T4199'

            $mockServiceObject = [PSCustomObject] @{
                StartupParameters = $mockStartupParameters
            }
        }

        Context 'When passing a service object' {
            It 'Should return the correct values' {
                $result = Get-SqlDscTraceFlag -ServiceObject $mockServiceObject

                $result | Should -HaveCount 1
                $result | Should -Contain 4199
            }
        }
    }
}

File debug.ps1

Switch UseBreakpoints to $true to verify that the tests passes using breakpoints for code coverage.

$pesterConfig = New-PesterConfiguration -Hashtable @{
    CodeCoverage = @{
        Enabled = $true
        Path = './SqlServerDsc.psm1'
        OutputPath = './Pester_coverage.xml'
        # Tests fails if this is $false
        UseBreakpoints = $false
    }
    Run = @{
        Path = '*.Tests.ps1'
    }
}

$Error.Clear()
$ErrorView = 'DetailedView'

Invoke-Pester -Configuration $pesterConfig

$Error

Describe your environment

Pester version     : 5.4.0 C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1                       
PowerShell version : 7.3.2
OS version         : Microsoft Windows NT 10.0.22623.0

Possible Solution?

Sorry, unknown.

johlju commented 1 year ago

Forgot to add the actual error.

Exception             : 
    Type       : System.ArgumentOutOfRangeException
    Message    : Index was out of range. Must be non-negative and less than or equal to the size of the collection. (Parameter 'index')
    ParamName  : index
    TargetSite : 
        Name          : Insert
        DeclaringType : System.Text.StringBuilder
        MemberType    : Method
        Module        : System.Private.CoreLib.dll
    Data       : System.Collections.ListDictionaryInternal
    Source     : System.Private.CoreLib
    HResult    : -2146233086
    StackTrace : 
   at System.Text.StringBuilder.Insert(Int32 index, String value)
   at System.Management.Automation.Language.PositionUtilities.BriefMessage(IScriptPosition position)
   at System.Management.Automation.ScriptDebugger.TraceLine(IScriptExtent extent)
   at System.Management.Automation.ScriptDebugger.OnSequencePointHit(FunctionContext functionContext)
   at System.Management.Automation.ScriptDebugger.EnterScriptFunction(FunctionContext functionContext)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
   at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockClauseToInvoke clauseToInvoke, Boolean createLocalScope, Dictionary`2 functionsToDefine, List`1 variablesToDefine, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Object[] args)
   at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Boolean propagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 functionsToDefine, Object[] args)
   at System.Management.Automation.ScriptBlock.InvokeAsMemberFunction(Object instance, Object[] args)
   at System.Management.Automation.Internal.ScriptBlockMemberMethodWrapper.InvokeHelper(Object instance, Object sessionStateInternal, Object[] args)
   at CallSite.Target(Closure, CallSite, Type)
   at System.Management.Automation.Interpreter.DynamicInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
CategoryInfo          : OperationStopped: (:) [], ArgumentOutOfRangeException
FullyQualifiedErrorId : System.ArgumentOutOfRangeException
InvocationInfo        : 
    ScriptLineNumber : 9
    OffsetInLine     : 9
    HistoryId        : -1
    ScriptName       : C:\source\DebugCodeCoverage\SqlServerDsc.psm1
    Line             : $startupParameters = [StartupParameters]::new()

    PositionMessage  : At C:\source\DebugCodeCoverage\SqlServerDsc.psm1:9 char:9
                       +         $startupParameters = [StartupParameters]::new()
                       +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    PSScriptRoot     : C:\source\DebugCodeCoverage
    PSCommandPath    : C:\source\DebugCodeCoverage\SqlServerDsc.psm1
    CommandOrigin    : Internal
ScriptStackTrace      : at Parse, C:\source\DebugCodeCoverage\SqlServerDsc.psm1: line 9
                        at Get-SqlDscTraceFlag, C:\source\DebugCodeCoverage\SqlServerDsc.psm1: line 45
                        at <ScriptBlock>, C:\source\DebugCodeCoverage\SqlServerDsc.Tests.ps1: line 22
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1998
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1959
                        at Invoke-ScriptBlock, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 2120
                        at Invoke-TestItem, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1194
                        at Invoke-Block, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 830
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 888
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1998
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1959
                        at Invoke-ScriptBlock, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 2123
                        at Invoke-Block, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 935
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 888
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1998
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1959
                        at Invoke-ScriptBlock, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 2123
                        at Invoke-Block, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 935
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 888
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1998
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1959
                        at Invoke-ScriptBlock, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 2123
                        at Invoke-Block, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 935
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 888
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1998
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1959
                        at Invoke-ScriptBlock, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 2123
                        at Invoke-Block, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 935
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1672
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.ps1: line 3
                        at <ScriptBlock>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 3164
                        at Invoke-InNewScriptScope, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 3171
                        at Run-Test, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 1675
                        at Invoke-Test, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 2475
                        at Invoke-Pester<End>, C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1: line 5272
                        at <ScriptBlock>, C:\source\DebugCodeCoverage\debug.ps1: line 17
                        at <ScriptBlock>, <No file>: line 1
fflaten commented 1 year ago

Thanks for the report. This is a weird one. Can reproduce, but have no idea what's going on. Will probably need to debug pwsh, so leaving it to our CC-wizard 🧙‍♂️

fflaten commented 1 year ago

This is also a PowerShell-bug. The invoked code isn't modified by us at all. Our injected code gets triggered right after this actually.

Repro:

class MyClass {
    # Any method has to exist, regardless of scriptblock, parameters or static/non-static
    MyMethod() { }
}

Set-PSDebug -Trace 1
[MyClass]::new()

Found this issue when I went to report it: https://github.com/PowerShell/PowerShell/issues/16874

fflaten commented 1 year ago

@johlju: I found a workaround for now. Add a constructor to the class, ex. StartupParameters() { }

johlju commented 1 year ago

Thank you for putting the work in, and also finding a workaround for this! It seems to work perfectly. I have re-enabled Pester's new code coverage method in the pipeline for SqlServerDsc.