Container is reported as Passed/Successful when discovery fails

fflaten commented 2 months ago

Checklist

[X] Issue has a meaningful title
[X] I have searched the existing issues. See all issues
[X] I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

When Discovery fails in a container after a valid test:

The blocks and tests up until the error is executed. Is this expected?
The block containing the discovery error is considered Passed
The container is reported as Passed even though it's listed under FailedContainers

The run is still reported as Failed and the error code returned from Invoke-Pester will be non-zero due to the failed container (set to 1 in repro).

Expected Behavior

Is this expected behavior? Should Container and block be successful when only partly discovered?

Steps To Reproduce

$sb = {
    Describe "d" {
        It '1' {}

        throw 'omg'
    }
}

$container = New-PesterContainer -ScriptBlock $sb
$r = Invoke-Pester -Container $container -PassThru

Describe your environment

Pester version : 6.0.0-alpha4 /workspaces/Pester/bin/Pester.psm1
PowerShell version : 7.4.1 OS version : Unix 5.15.153.1

Possible Solution?

No response

nohwnd commented 2 months ago

The blocks and tests up until the error is executed. Is this expected+

Yes, originally I was failing just after discovery, but then it was very hard to migrate tests gradually because all tests had to be discoverable, instead of going test by test, file by file.

The block containing the discovery error is considered Passed

I think this should fail the block.

The container is reported as Passed even though it's listed under FailedContainers

Same here the container should fail.

fflaten commented 2 months ago

Thanks, that makes sense.

So fix the result values, but keep the partial container execution as-is?

nohwnd commented 2 months ago

I think it makes sense that way, what do you think?

pester / Pester