pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

Filter.FullName / -FullNameFilter not applying to data driven tests #1891

Open o-o00o-o opened 3 years ago

o-o00o-o commented 3 years ago

General summary of the issue

Often I want to run a subset of my data driven tests - however it seems that I can't use -FullNameFilter for this as it doesn't apply to the post-processed name after expanding the hashtable values in <>'s

Describe your environment

Pester version : 5.1.1 C:...\Documents\WindowsPowerShell\Modules\pester\5.1.1\Pester.psm1 PowerShell version : 5.1.19041.610 OS version : Microsoft Windows NT 10.0.19041.0

Steps to reproduce

testing.ps1

BeforeAll {
  Function Get-Value {
    Write-Output 'blah'
  }
}

Describe "Get-Emoji <name>" -ForEach @(
    @{ Name = "cactus"; Kind = 'Plant' }
    @{ Name = "giraffe"; Kind = 'Animal' }
) {
    It "Returns <name>"  {
        Get-Value
    }
}

now run

> invoke-pester -Path .\test\suites\pestertesting\ -Output Detailed

Starting discovery in 1 files.
Discovering in testing.Tests.ps1.
Found 2 tests. 93ms
Discovery finished in 127ms.

Running tests from 'testing.Tests.ps1'
Describing Get-Emoji cactus
  [+] Returns cactus 32ms (2ms|30ms)

Describing Get-Emoji giraffe
  [+] Returns giraffe 9ms (1ms|8ms)
Tests completed in 610ms
Tests Passed: 2, Failed: 0, Skipped: 0 NotRun: 0

however if you run

invoke-pester -Path .\test\suites\pestertesting\ -FullNameFilter '*cac*'

Starting discovery in 1 files.
Discovery finished in 105ms.
Tests completed in 99ms
Tests Passed: 0, Failed: 0, Skipped: 0 NotRun: 2

Looking into the details of the not run you can see that the ExpandedName hasn't been expanded

> $blah = invoke-pester -Path .\test\suites\pestertesting\ -FullNameFilter '*cac*' -PassThru 
> $blah.NotRun

Name              : Returns <name>
Path              : {Get-Emoji <name>, Returns <name>}
Data              :
ExpandedName      : Returns <name>
ExpandedPath      : Get-Emoji <name>.Returns <name>
Result            : NotRun
ErrorRecord       : {}
StandardOutput    :
Duration          : 00:00:00
ItemType          : Test
Id                :
ScriptBlock       :
                            Get-Value

Tag               :
Focus             : False
Skip              : False
Block             : [ ] Get-Emoji <name>
First             : False
Last              : False
Include           : False
Exclude           : False
Explicit          : False
ShouldRun         : False
StartLine         : 11
Executed          : False
ExecutedAt        :
Passed            : False
Skipped           : False
UserDuration      : 00:00:00
FrameworkDuration : 00:00:00
PluginData        :
FrameworkData     :

Note that this same issue happens on describe and context blocks also. Nothing is expanded

Expected Behavior

I expect that test names/context/description etc are filtered after expansion

Current Behavior

no way to filter to a subset of data driven tests (!)

fflaten commented 3 years ago

This is due to filters being processed in Discovery-phase, while the fullname (path of block/test) has variables expanded during the Run-phase.

@nohwnd Any thoughts on how to solve this? Moving the variable expansion to discovery would break support for using variables from BeforeAll/-Each in test/block names. We could expand name/path twice, replacing only known variables from Data during Discovery (New-Block/Test) with a regex MatchEvaluator. However the filter might feel inconsistent as only some variables would be expanded.

Todo: The filter doesn't use ExpandedPath atm. Could probably replace .Path -join '.' with .ExpandedPath in multiple places.

o-o00o-o commented 3 years ago

Thanks for the response @fflaten

I'm currently hacking this with the following at the top of my It block, but it would be good to get this resolved by the framework itself

if ($____Pester.Configuration.Filter.FullName.Value -and $____Pester.CurrentTest.ExpandedPath -notlike $____Pester.Configuration.Filter.FullName.Value) {
    # work around the bug with pester
    Set-ItResult -Skipped -Because "We skipped the test due to filters on FullName. ($($____Pester.CurrentTest.ExpandedPath)) -notlike ($($____Pester.Configuration.Filter.FullName.Value))"
    break
}

However I found I couldn't use the same code in the BeforeEach. I didn't look into why that was too deeply - I just had to move the BeforeEach code out into the It block. At first glance it looked like the CurrentTest.ExpandedPath wasn't expanded in the BeforeEach either (perhaps a similar issue as you already identified)

o-o00o-o commented 3 years ago

A seperate issue I know but it would be good if we could also fix the purple text output when providing a FullName string/array. It always shows "Filter 'FullName' set to ('System.String[]')." and doesn't show the actual values. Tag works ok as you can see below

Starting discovery in 1 files.
Filter 'FullName' set to ('System.String[]').
Filter 'Tag' set to ('LOAD_FROM').
fflaten commented 3 years ago

A seperate issue I know but it would be good if we could also fix the purple text output when providing a FullName string/array. It always shows "Filter 'FullName' set to ('System.String[]')." and doesn't show the actual values. Tag works ok as you can see below

Starting discovery in 1 files.
Filter 'FullName' set to ('System.String[]').
Filter 'Tag' set to ('LOAD_FROM').

Moved to separate issue. 🙂

nohwnd commented 3 years ago

@fflaten Yes that is exactly why it does not work. Filtering is and mostly likely always will be Discovery only concept. This is a tax for being able to expand variables from the current scope into the name.

I am also considering making the -ForEach have a lazy version so you can evaluate the testcases during Run and not during Discovery, this would allow using BeforeAll variables in testcases (but not BeforeEach), and it would further complicate the way tests are run and filtered... So balancing all of this is difficult.

I still would like to have a -Filter where you can provide a scriptblock predicate and have it filter on whatever, but that would have to be intelligent about this and evaluated during discovery as well as during Run, but not in both.

In short I don't see any simple solution for this at the moment, unless ExpandedFullName filter is added. It seems that filtering will be best kept as Discovery only concept and skipping will be used to skip the current test during run.

As @o-o00o-o shown above maybe Set-ItResult could get a -When parameter that takes a predicate and filters on the current object.

The hacky code seems to not work for BeforeAll because .CurrentBlock should be used instead of current test, and the test also does not re-check $skip on itself of the parent block when set this late.

But overall it seems doable.

nohwnd commented 3 years ago

Moving this to next milestone because it requires a new cmdlet to be published that would be tied to the internal state of Pester, and this requires thinking about how to best present that data, we probably should wrap the internal object into another object to protect it from being changed in the future. And discuss the shape of that new object.