Error in BeforeAll of Data driven tests leads to missing info in the error output

crlgb commented 1 month ago

Checklist

[X] Issue has a meaningful title
[X] I have searched the existing issues. See all issues
[X] I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

Output does not resolve the current item $_ in the output.

Starting discovery in 1 files.
Discovery found 2 tests in 2ms.
Running tests.
[-] Describe Test <_> failed
 RuntimeException: something
[...]
Tests completed in 24ms
Tests Passed: 0, Failed: 2, Skipped: 0, Inconclusive: 0, NotRun: 0
BeforeAll \ AfterAll failed: 2
  - Test <_>
  - Test <_>

Expected Behavior

Starting discovery in 1 files.
Discovery found 2 tests in 2ms.
Running tests.
[-] Describe Test **a** failed
 RuntimeException: something
[...]
Tests completed in 24ms
Tests Passed: 0, Failed: 2, Skipped: 0, Inconclusive: 0, NotRun: 0
BeforeAll \ AfterAll failed: 2
  - Test **a**
  - Test **b**

Steps To Reproduce

Describe "Test <_>" -ForEach @("a","b") {
  BeforeAll {
    throw "something"
  }
  It "is a or b" {
    $_ | Should -Match "[ab]"
  }
}

Describe your environment

Pester version     : 5.6.1
PowerShell version : 7.4.1
OS version         : Unix 15.0.1

Possible Solution?

No response

nohwnd commented 1 month ago

This is by design. We evaluate the name just after BeforeEach, so values from BeforeAll and BeforeEach are reflected into the string. This seemed like the best place, since the Before* steps are setting up values and should hopefully have all the right data for the test.

We also need to evaluate in a scope that is on the same level as the BeforeEach.

The code here could probably be changed to capture the error from BeforeEach, run more code to evaluate the name of the test, and rethrow the error.

https://github.com/pester/Pester/blob/main/src/Pester.Runtime.ps1#L643

fflaten commented 1 month ago

This is known limitation and currently by design.

The variables are expanded after setup (BeforeAll/-Each) to have all variables/reference available. An exception during setup stops further processing to avoid unexpected results like old variables from parent scope being used etc.

@nohwnd Candidate for v6? Does anyone depend on BeforeEach/-All variables for templates?

nohwnd commented 1 month ago

IDK if we technically we can fix it.

There can also be a different complaint which is: the variables from my tests / AfterAll are not expanded.

neo42JBR commented 3 days ago

Chiming in on this topic:

I built a framework that can parse configuration files to pester tests. In it there is an option to specifiy if a test is terminating or not. As you can use Set-ItResult (or its errorrecord) within BeforeEach the tests get the correct state but their title is not set correctly.

The variables might still be present (especially when coming from the testdata) and one could atleast try to expand them.

Edit: Another case in which the above happens is when the Test is skipped via the -Skip parameter

pester / Pester