Failure in Discovery when using different parameters across files

GSDragoon commented 1 year ago

Checklist

[X] Issue has a meaningful title
[X] I have searched the existing issues. See all issues
[X] I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

I have a suite of tests across multiple files that are the target of a single Pester Container definition that is used to pass parameters into the tests. The path of the container uses a wildcard to target all the test files. Not all of the test files use the same parameters. The parameters also make use of the Parameter attribute (such as forcing it to be mandatory). Manually building a test container for each combination isn't really feasible.

When I run the tests, the discovery fails for the files that do not have all the parameters (and make use of the Parameter attribute).

Pester v5.5.0
Discovery: Starting test discovery in 5 test containers. 

Starting discovery in 5 files.
Discovery: Discovering tests in C:\s\Local\PesterBug\NotWorking1.Tests.ps1 
[-] Discovery in C:\s\Local\PesterBug\NotWorking1.Tests.ps1 failed with:
System.Management.Automation.ParameterBindingException: A parameter cannot be found that matches parameter name 'Parameter2'.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at Invoke-File(Closure , FunctionContext )
   at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProcess)
   at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
   at System.Management.Automation.CommandProcessorBase.Complete()
at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 3034
at Invoke-File, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 3043
at Invoke-BlockContainer, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 2942
at Discover-Test, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1468
at Invoke-Test, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 2462
at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 5046
at <ScriptBlock>, C:\s\Local\PesterBug\RunTests.ps1: line 19
at <ScriptBlock>, <No file>: line 1
Discovery: Found 0 tests in 15 ms 
Discovery: Discovering tests in C:\s\Local\PesterBug\NotWorking2.Tests.ps1 
[-] Discovery in C:\s\Local\PesterBug\NotWorking2.Tests.ps1 failed with:
System.Management.Automation.ParameterBindingException: A parameter cannot be found that matches parameter name 'Parameter2'.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at Invoke-File(Closure , FunctionContext )
   at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProcess)
   at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
   at System.Management.Automation.CommandProcessorBase.Complete()
at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 3034
at Invoke-File, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 3043
at Invoke-BlockContainer, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 2942
at Discover-Test, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 1468
at Invoke-Test, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 2462
at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1: line 5046
at <ScriptBlock>, C:\s\Local\PesterBug\RunTests.ps1: line 19
at <ScriptBlock>, <No file>: line 1
Discovery: Found 0 tests in 15 ms 
Discovery: Discovering tests in C:\s\Local\PesterBug\Working1.Tests.ps1 
Discovery: Found 1 tests in 12 ms 
Discovery: Discovering tests in C:\s\Local\PesterBug\Working2.Tests.ps1 
Discovery: Found 1 tests in 15 ms 
Discovery: Discovering tests in C:\s\Local\PesterBug\Working3.Tests.ps1 
Discovery: Found 1 tests in 12 ms 
Discovery: Processing discovery result objects, to set root, parents, filters etc. 
Discovery found 3 tests in 96ms.
Discovery: Test discovery finished. 
Running tests.

Running tests from 'C:\s\Local\PesterBug\Working1.Tests.ps1'
Describing Two parameters with parameter and validate attributes
  [+] Should pass 22ms (14ms|7ms)

Running tests from 'C:\s\Local\PesterBug\Working2.Tests.ps1'
Describing One parameter with validate attribute
  [+] Should pass 21ms (8ms|13ms)

Running tests from 'C:\s\Local\PesterBug\Working3.Tests.ps1'
Describing One parameter with no attributes
  [+] Should pass 17ms (10ms|8ms)
Tests completed in 480ms
Tests Passed: 3, Failed: 0, Skipped: 0 NotRun: 0
Container failed: 2
  - C:\s\Local\PesterBug\NotWorking1.Tests.ps1
  - C:\s\Local\PesterBug\NotWorking2.Tests.ps1

Expected Behavior

I would expect the tests files that do not have all the parameters to work. They should ignore the extra parameters like how it works if the Parameter attribute is not applied to the parameter.

Steps To Reproduce

Commands to build the Pester Container, Configuration and run the tests.

$container = New-PesterContainer -Path '*.Tests.ps1' -Data @{
    Parameter1 = 'foo'
    Parameter2 = 'bar'
}

$config = New-PesterConfiguration @{
    Run = @{
        Container = $container
    }
    Output = @{
        Verbosity = 'Diagnostic'
        StackTraceVerbosity = 'Full'
    }
    Debug = @{
        WriteDebugMessages = $true
    }
}

Invoke-Pester -Configuration $config

Working test files.

param (
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $Parameter1,
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $Parameter2
)

Describe 'Two parameters with parameter and validate attributes' {
    It 'Should pass' {
        $Parameter1 | Should -Be 'foo'
        $Parameter2 | Should -Be 'bar'
    }
}

param (
    [ValidateNotNullOrEmpty()]
    [string] $Parameter1
)

Describe 'One parameter with validate attribute' {
    It 'Should pass' {
        $Parameter1 | Should -Be 'foo'
    }
}

param (
    [string] $Parameter1
)

Describe 'One parameter with no attributes' {
    It 'Should pass' {
        $Parameter1 | Should -Be 'foo'
    }
}

Test files that do not work.

param (
    [Parameter()]
    [string] $Parameter1
)

Describe 'One parameter with parameter attribute' {
    It 'Should pass' {
        $Parameter1 | Should -Be 'foo'
    }
}

param (
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $Parameter1
)

Describe 'One parameter with parameter and validate attribute' {
    It 'Should pass' {
        $Parameter1 | Should -Be 'foo'
    }
}

Describe your environment

Pester version : 5.5.0 C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1 PowerShell version : 5.1.19041.3031 OS version : Microsoft Windows NT 10.0.19045.0

Possible Solution?

There are two options I see for workarounds, neither are that great, but not terrible.

Define and set a default value of null for parameters that aren't used in those files.

param (
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $Parameter1,
    [string] $Parameter2 = $null
)

Have one file that meets the Path criteria that defines all the parameters. Then it dot sources the other test files (that do not meet the Path criteria) that do not have any parameters defined.

param (
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $Parameter1,
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $Parameter2
)

Describe 'My Tests'  {
    . "$PSScriptRoot\File1.ps1"
    . "$PSScriptRoot\File2.ps1"
    ...
    . "$PSScriptRoot\File30.ps1"
}

fflaten commented 1 year ago

Hi. This is intended PowerShell behavior and not exclusive to Peater. When you use [Parameter()] or [CmdletBinding()] it becomes an advanced function/script. Arguments are then disabled by default so if you try to splat or call with an undefined parameter it will throw.

You need to provide Data compatible with every script or use multiple calls to New-PesterContainer with the appropriate Data values per script.

johlju commented 1 year ago

@GSDragoon does it work to use ValueFromRemainingArguments?

GSDragoon commented 1 year ago

Hi. This is intended PowerShell behavior and not exclusive to Peater. When you use [Parameter()] or [CmdletBinding()] it becomes an advanced function/script. Arguments are then disabled by default so if you try to splat or call with an undefined parameter it will throw.

You need to provide Data compatible with every script or use multiple calls to New-PesterContainer with the appropriate Data values per script.

Thanks for the quick response. Guess I shouldn't be too surprised this is the case. Sounds like I need to be okay with a workaround or make bigger design changes

@GSDragoon does it work to use ValueFromRemainingArguments?

Yes, this works. Suppose this is another workaround, to add this parameter to each file.

param (
    [Parameter()]
    [string] $Parameter1,
    [Parameter(ValueFromRemainingArguments)]
    [object[]]$RemainingParametersPlaceholder
)

pester / Pester