pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

v5.5 $____Pester.CurrentTest not populated in AfterEach #2409

Closed Scal-Human closed 10 months ago

Scal-Human commented 11 months ago

Checklist

What is the issue?

Reviewing tests for multiple module, I wanted to have a verbose trace during Pester analysis after each test. I created a context within which an AfterEach block is defined and I am expecting the $Pester.CurrentTest to contain test information after its execution (Passed or Status, Duration), but the $Pester.CurrentTest is partialy populated (Path is ok, Executed is ok, but Passed is always false).

Disclaimer: I was unable to find the documentation about $____Pester

Expected Behavior

Ok : First test gives the Pester version and is meant to succeed KO; AfterEach should show Status , Duration, and Passed true but it is false Ok: Second test is supposed to explode as it uses an unexisting drive KO; AfterEach should show Status , Duration but are missing.

Steps To Reproduce

Note: I also tried by having having Context inside Describe with the same result.

[CmdletBinding()]
Param ()
Context 'Pester AfterEach Issue' {
    BeforeEach {
        Write-Verbose ('BeforeEach: {0} {1}' -f ('-' * 8), ($____Pester.CurrentTest.Path -Join ' / '))
    }
    AfterEach {
        Write-Verbose ('AfterEach : Executed {0}, Passed {1}, Status {2}, Duration {3}' -f
            $____Pester.CurrentTest.Executed, $____Pester.CurrentTest.Passed,
            $____Pester.CurrentTest.Status, $____Pester.CurrentTest.Duration
        )
    }

    Describe 'Pester' {
        It 'Has version' {
            $version = (Get-Module Pester).Version
            Write-Verbose ('Pester : {0}' -f $version)
            $version | Should -Not -Be $Null
        }
    }
    Describe 'New-Item' {
        It 'Should explode' {
            New-Item Dummy:\Boom -Type Explode | Should -Not -Be $Null
        }
    }
}

outputs:

Starting discovery in 1 files.
Discovery found 2 tests in 31ms.
Running tests.
VERBOSE: BeforeEach: -------- Pester AfterEach Issue / Pester / Has version
VERBOSE: Pester : 5.5.0
VERBOSE: AfterEach : Executed True, Passed False, Status , Duration 00:00:00
VERBOSE: BeforeEach: -------- Pester AfterEach Issue / New-Item / Should explode
New-Item: C:\Project\Vulcan\Tangible\Source\Pester-Issue.ps1:23
Line |
  23 |              New-Item Dummy:\Boom -Type Explode | Should -Not -Be $Nul …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Cannot find drive. A drive with the name 'Dummy' does not exist.
VERBOSE: AfterEach : Executed True, Passed False, Status , Duration 00:00:00
[-] Pester AfterEach Issue.New-Item.Should explode 33ms (32ms|1ms)
 Expected $null to be different from the actual value, but got the same value.
 at New-Item Dummy:\Boom -Type Explode | Should -Not -Be $Null, C:\Project\Vulcan\Tangible\Source\Pester-Issue.ps1:23
 at <ScriptBlock>, C:\Project\Vulcan\Tangible\Source\Pester-Issue.ps1:23
Tests completed in 113ms
Tests Passed: 1, Failed: 1, Skipped: 0 NotRun: 0

results are correct:

Name           Path                                               Data ExpandedName   ExpandedPath                                   Result ErrorRecord
----           ----                                               ---- ------------   ------------                                   ------ -----------
Has version    {Pester AfterEach Issue, Pester, Has version}           Has version    Pester AfterEach Issue.Pester.Has version      Passed {}
Should explode {Pester AfterEach Issue, New-Item, Should explode}      Should explode Pester AfterEach Issue.New-Item.Should explode Failed {Expected $null to be different from the actual value, but got the same value.}

Describe your environment

I am running on Windows 10 with PowerShell core 7.3.10.

Possible Solution?

No response

fflaten commented 11 months ago

Thanks for the report. This is the expected behavior as BeforeEach and AfterEach is considered part of the test, so properties in $____Pester.CurrentTest isn't updated at all until all three have been executed.

Btw. $____Pester is intentionally undocumented as it's a private API. Once a public API is provided in the future (#318) for these types of scenarios it might behave differently, ex updated per stage.

Scal-Human commented 11 months ago

Thanks for the response. So this is suppose to work this way ? This make $____Pester just useless and no possible way (other than custom trick) to get test status information about current test in the AfterEach (which is after the test and not part of the test). Exact ? The fact that the Path to the test is correct makes us think this variable is populated. Why no just remove it ? Not working + not documented + no intention to make it work = confusing situation.

Thanks, anyway

fflaten commented 10 months ago

... get test status information about current test in the AfterEach (which is after the test and not part of the test). Exact ?

Currently in Pester v5 BeforeEach and AfterEach is considered part of the test and are executed in a single execution. The CurrentTest object is updated before this execution and after (e.g. determine Result).

We could compare it to an school exam:

  1. You grab your answer sheet and pencil (BeforeEach)
  2. You fill it out (Test/It-block)
  3. You deliver the answer sheet (AfterEach)

If you fail at any of these steps, you've failed the exam. So until all three steps are complete it's too early to determine the final Result.

I fully agree that being able to get the current Result during AfterEach is a special case, as it may help you collect more data for troubleshooting. That particular scenario is tracked in #1821 which is pending #318 for a way to expose the information.

Whether a failure in AfterEach should modify the final Result is another discussion. @nohwnd Do you know how teardown errors behave in other test frameworks?

The fact that the Path to the test is correct makes us think this variable is populated. Why no just remove it ? Not working + not documented + no intention to make it work = confusing situation.

As mentioned it is a private API, intentionally given a weird variable name and being undocumented, to avoid users from depending on it.

While not being useful for end users, it still brings us value during troubleshooting, debugging and figuring out which parts of it that should be exposed in the final public API (#318).

Scal-Human commented 10 months ago

Hello, Many thanks for this detailed answer.

I close the issue as it wil not be solved, and I use a different approach anyway.

Have a nice weekend.