Add "Context" / "It" level granularity to -TestName parameter

[vors]

Current granularity allows only Describe block level granularity.

One important dev scenario:

There is a pester suite with one Describe block, that runs long time (minutes). I have a failure in one It. I want to be able modify my production code, run / debug test (without modifying test code) as fast as possible (seconds). I'm only interested in one It.

Currently, I have to modify test code: comment all but one tests.

[vors]

Proposed syntax: Invoke-Pester -TestName "DescribeBlockName[\ContextBlockName[\ItBlockName]]"

[nohwnd]

@vors That is an interesting idea, I also miss it sometimes myself. I have some questions: 1) Describe and context names are usually pretty verbose, copy pasting them in console is a bit clumsy. Is there another syntax you considered? 2) I suppose the before each, after each and other setup code, including all the code placed in describes should run. How would you implement this? As a simple if on the names? 3) should wildcards be supported?

[dlwyatt]

We already support wildcards in the Describe portion, so may as well keep that behavior. Then you could do something like -TestName '*PartialDescribeName*\*PartialContextName*\*Whatever*'. We'd just split the string on the backslashes and take it from there.

It would be nice to be able to just specify the 'Whatever' part and have Pester figure out that there's a deep nested test that needs to be run, but since we're executing the code as it's being read (rather than parsing ahead of time), that's not really an option at the moment.

[nohwnd]

Should \\*correcly* run all the tests that include "correctly" in its name?

[dlwyatt]

That, or '*\*\*correctly*'. However, I see that as being of pretty limited use right now. Most of the time spent is out in the Describe and Context blocks anyway, and they'd still be executed, even if there were no matching It blocks inside them.

[vors]

@nohwnd 1) I think it's fine: at least way better then existing alternative. 2) yes, this is exactly what I had in mind: simple if on name. 3) Dave already covered that, I personally don't see value in wildcards, except '*\*\*correctly*'

One think that I missed: what to do if Describe block name has \ inside. I think its' a corner case and should not stop us from implementing, but I don't have a good answer from the top of my head.

[nohwnd]

Just thinking out loud... Trying look a bit ahead I am afraid this syntax won't play nice with the planned ability to have arbitrary number of child suites. And using AST to parse the whole script (or all of them if file is not specified) seems like overkill for such feature, not to mention it would be only PowerShell 3+ feature.

[vors]

What's our options for syntax? I can think only about -TestsScript "xxx<delimiter>yyy<delimiter>zzz". New parameter, hashtable, collection instead of string look like overkill.

[dlwyatt]

I was thinking about that, and the current proposed syntax is probably fine. We already run into some ambiguity when the filter looks like this: 'Level1\Level2'. Is Level2 a Context or an It? Could be either one, and if Pester finds a matching name, it'll probably just run whatever that match happens to be. The same should be true once there's arbitrary levels of nesting.

[ghostsquad]

+1 if you want the AST code to parse, I could probably cook that up for you. It's something I've been working on for project that has a different approach to Powershell testing (PSClasses & Xunit/NUnit style test cases): https://github.com/ghostsquad/AetherUnit

You can look at the AST parsing code that I've been working on here:

https://github.com/ghostsquad/AetherUnit/blob/dev2-ast/Module/PSClasses/GpunitFixtureDiscoverer.ps1

I'm commenting here, because this request is one feature I definitely need to keep my testing running as fast as possible. In addition, AST would solve the problem of needed to load every test file in order to find out what tests are inside of it. My test files contain boilerplate code like below that make "targeted" test runs pretty slow.

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path).Replace(".Tests.", ".")
$private:moduleRoot = Resolve-Path (Join-Path $here "..\..")
. (Join-Path $moduleRoot 'TestCommon.ps1')
. ([System.IO.Path]::Combine($moduleRoot, $sut))
$fakesDir = Join-Path $global:TestRootDirectory "fakes"

Describe ...

[dlwyatt]

We can't rely on the AST for now, or Pester will no longer work properly for PowerShell 2.0. At some point down the road when we stop supporting that version, then we can do all sorts of neat stuff that uses the AST.

[nohwnd]

I touched on this in #1265 and I am not sure if \ or . is the separator to choose. To me it would avoid confusion between tests file path (*.Tests.ps1) and a "test" path (Get-Path\uses current directory as the default path), and would be similar to .net types and their fully qualified names. But I can also see how it is useful to always delimit by just one thing, and in that case \ is probably better.

Circling back to #1048 this does not solve the issue with the syntax being ambiguous, but for that I have other tools. Internally in v5 I handle the test path as collection of strings. Joining them by the separator and comparing them via like would make this work. And the syntax would be quite simple, just a string / string array:

@('*\test2', '*\describe2\*\test*')

Now the question is, is this still useful? :) Because you can simply put -Focus on the failing test and run just that. But I guess it would avoid the need for tags in some cases? But does that warrant explaining what a test-path is and dispelling the confusion from a tests file path?:D

[vors]

Oh so the proposed solution is to add a unique new tag and run with -Focus TagName ?

[splatteredbits]

I've submitted a PR that adds -ItFilter and -ContextFilter parameters to Invoke-Pester. This should work in PowerShell 2. https://github.com/pester/Pester/pull/1374

[maxim-lobanov]

Please finalize PR above, that is the great improvement for our use-case too

[nohwnd]

@maxim-lobanov I am sorry but I won't (as explained in the PR, and the article linked from that PR). There are many edge cases to this, and I don't want to change the api that will be frozen with the next version.