SkipRemainingOnFailure does not function correctly when there are nested BeforeAll blocks

[datacore-bwynne]

Checklist

• [X] Issue has a meaningful title
• [X] I have searched the existing issues. See all issues
• [X] I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

Pester v5.3.3 Invoke Pester with: $config.Run.SkipRemainingOnFailure = "Container"

If there are two describe blocks (A & B) that have nested BeforeAll steps If the first Describe block(A) fails and hits and error. The following desctibe block(B) should be completely skipped. The issue is that the BeforeAll of the second Describe Block (B) is executed.

Expected Behavior

If there are two describe blocks(A & B) that have nested BeforeAll steps If the first Describe block(A) fails and hits and error. The following describe block(B) should be completely skipped. Specifically: The BeforeAll of the second Describe Block (B) should NOT be executed.

Steps To Reproduce

No response

Describe your environment

The following works - Example 1 without BeforeAll in Describe B: Example 1 Code:

Describe "A"{
    BeforeAll {
        Write-Host "Before ALL -A"
    }

    It "Test 1"{
        Write-Host "This is test 1"
        $a = 1
        $b = 2

        $a | Should -BeExactly $b
    }
}

Describe "B"{
     It "Test 2"{
        Write-Host "This is test 2"
    }
}

Example 1 - Output - without BeforeAll in Describe B:

Running tests from 'C:\trial.Tests.ps1'
Before ALL -A
Describing A
This is test 1
  [-] Test 1 26ms (19ms|7ms)
   Expected exactly 2, but got 1.
   at $a | Should -BeExactly $b, C:\trial.Tests.ps1:11
   at <ScriptBlock>, C:\trial.Tests.ps1:11

Describing B
Skip: (B.Test 2) Test is skipped. 
  [!] Test 2 10ms (0ms|10ms)
Tests completed in 264ms
Tests Passed: 0, Failed: 1, Skipped: 1 NotRun: 0
Remaining tests skipped after first failure: 1

The following does not run as expected: Example 2- code - with BeforeAll in Describe B

Describe "A"{
    BeforeAll {
        Write-Host "Before ALL -A"
    }

    It "Test 1"{
        Write-Host "This is test 1"
        $a = 1
        $b = 2

        $a | Should -BeExactly $b
    }
}

Describe "B"{
    BeforeAll {
        Write-Host "Before ALL -B"
    }

     It "Test 2"{
        Write-Host "This is test 2"
    }
}

Example 2: output - with BeforeAll in Describe B

Running tests from 'C:\trial.Tests.ps1'
Before ALL -A
Describing A
This is test 1
  [-] Test 1 30ms (21ms|8ms)
   Expected exactly 2, but got 1.
   at $a | Should -BeExactly $b, C:\trial.Tests.ps1:11
   at <ScriptBlock>, C:\trial.Tests.ps1:11
Before ALL -B

Describing B
Skip: (B.Test 2) Test is skipped. 
  [!] Test 2 11ms (0ms|11ms)
Tests completed in 288ms
Tests Passed: 0, Failed: 1, Skipped: 1 NotRun: 0
Remaining tests skipped after first failure: 1

Possible Solution?

No response

[fflaten]

Thanks, good catch! Looks like the feature only skipped tests, not blocks, so all BeforeAll/AfterAll will be invoked in both nested and sibling blocks. Will fix this.

[billgothacked]

Don't forget what it said in "Black Hat Python" about one of the ways you can get caught. Repositories! I don't like when people fuck with my family and I don't like perv's listening. Peace

On Sun, Mar 31, 2024 at 3:28 AM Frode Flaten @.***> wrote:

Thanks, good catch! Looks like the feature only skipped tests, not blocks, so all BeforeAll/AfterAll will be invoked in both nested and sibling blocks. Will fix this.

— Reply to this email directly, view it on GitHub https://github.com/pester/Pester/issues/2424#issuecomment-2028629479, or unsubscribe https://github.com/notifications/unsubscribe-auth/BA7FZAVOOTYKGZNHKHGPXHTY27QMNAVCNFSM6AAAAABDTUTRCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGYZDSNBXHE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[datacore-bwynne]

Hello @fflaten, Will the following 'Bug scenario 2' also be fixed?

If the first describe block fails, with using the SkipRemainingOnFailure, I expected the second describe block to NOT execute.

Bug Scenario 2

Describe "TestA"{
    BeforeAll{
         throw "BeforeAll Failure in TestA"
    }
   It "TestA1"{
       Write-Host "Test A1 - It"
   }
}

Describe "TestB" {
    BeforeEach{
        Write-Host "TestB - before all"
    }
    It "TestB2"{
        Write-Host "Test B2 - It"
    }
}

Output:

Describing TestA
[-] Describe TestA failed
  RuntimeException: BeforeAll Failure in TestA
  at <ScriptBlock>, C:\example.Tests.ps1:6

Describing TestB
TestB - before all
Test B2 - It
  [+] TestB2 102ms (86ms|16ms)
Tests completed in 569ms
Tests Passed: 1, Failed: 1, Skipped: 0 NotRun: 0
BeforeAll \ AfterAll failed: 1
  - TestA

[fflaten]

Not at the moment, but maybe it should. 🙂 Moved to a new issue for now.

The feature was designed for a failed test, while your scenario 2 is a block failure where further processing is always stopped for that block.