[Question] How to distinguish a failure and an error ?

[LaurentDardenne]

Can I use this code for that ?

$Result=Invoke-Pester -Passthru -Show None -Strict

#Failure
$Result.TestResult.ErrorRecord.FullyQualifiedErrorId -eq 'PesterAssertionFailed'

#Error
$Result.TestResult.ErrorRecord.FullyQualifiedErrorId -neq 'PesterAssertionFailed'

Versions :

Pester  4.3.1

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.15063.1155
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0, 5.0, 5.1.15063.1155}
BuildVersion                   10.0.15063.1155
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

[it-praktyk]

Can you try to explain what would you like achieve?

[nohwnd]

That really depends on your definition of Error and Failure. From the code snippet I assume that you want to distinguish between failing the test in an assertion, and in some random exception. For that you can use the fully qualified error id, as long as you are using the built in assertions. You can also use the Verify-AssertionFailed axiom.

Now my question also is, what are you trying to do?:)

[LaurentDardenne]

@it-praktyk 'Can you try to explain what would you like achieve?' The usage context is relative to Operation-Validation-Framework.

In my tests I can have syntax errors in the code or non-existent command calls.I know that a test fails or passes and that there is no intermediate state. If I look at the detail :

$r=invoke-pester -PassThru
$r.TestResult|select result,passed,@{Name='FullyQualifiedErrorId';e={$_.errorrecord.FullyQualifiedErrorId}},failuremessage

Result       Passed FullyQualifiedErrorId                           FailureMessage
------       ------ ---------------------                           --------------
Failed        False UnexpectedToken                                 Au caractère C:\Users\????\Demo.Tests.ps1:8 : 5...
Failed        False No module named 'MyModule' is currently loaded. No module named 'MyModule' is currently loaded.
Failed        False PesterAssertionFailed                           Expected $true, but got $false.
Inconclusive  False PesterTestInconclusive                          I'm inconclusive because I can.
Failed        False PesterAssertionFailed                           Expected $true, but got $false.
Failed        False CommandNotFoundException                        The term "shuld" is not recognized as a cmdlet name, function,

@nohwnd 'what are you trying to do?' I would like to differentiate failed tests :

Result       Passed FullyQualifiedErrorId                           FailureMessage
------       ------ ---------------------                           --------------
Failed        False PesterAssertionFailed                           Expected $true, but got $false.

of those that contain errors.

Result       Passed FullyQualifiedErrorId                           FailureMessage
------       ------ ---------------------                           --------------
Failed        False UnexpectedToken                                 Au caractère C:\Users\????\Demo.Tests.ps1:8 : 5...

In the latter case the use of Verify-AssertionFailed does not help.

[nohwnd]

@LaurentDardenne in that case you can add another layer that will filter the exceptions and inform you when the test failed outside of an assertion. But generally that is still a failed test.

function AssertionOrError ([ScriptBlock]$ScriptBlock) {
    try {
            $null = & $ScriptBlock
    }
    catch [Exception]
    {
        $assertionExceptionThrown = ($_.FullyQualifiedErrorId -eq 'PesterAssertionFailed')
        if (-not $assertionExceptionThrown){
            throw New-Object -TypeName Exception -ArgumentList "Test failed due to an error. $($_.Exception.Message)", $_.Exception
        }

        throw $_
    }
}

Describe "a" {
    It "passes" { 
        AssertionOrError { 
            $true | Should -Be $true
        }
    }

    It "fails in assertion" { 
        AssertionOrError { 
            $true | Should -Be $false
        }
    }

    It "fails in other place" { 
        AssertionOrError { 
            throw "Something bad happened!"
        }
    }
}

[LaurentDardenne]

@nohwnd But generally that is still a failed test. I agree, it is a diversion of pester.

Thanks to you two.

[LaurentDardenne]

There may be many of us who want to make Pester a rule engine :-)

[nohwnd]

@LaurentDardenne there are few other projects that I am aware of. But you still did not tell me how do you use the "failed in assertion" and "failed outside of assertion" differentiation. :) There could be another one, "failed in framework", but they all will become red on the server, and inherently a thing that you need to take action about.

[LaurentDardenne]

I call Pester with tests (1-n) :

function Test-SyntaxeShuld {
    [CmdletBinding()]
    param ( 
        [Parameter(Mandatory)]
        [ValidateNotNullOrEmpty()]
        [string]$VmName
     )
  try {
    $Pester=Invoke-Pester -Script @{Path="$PSScriptRoot\Rules\SyntaxeShuld.Tests.ps1";Parameters = @{VmName=$VmName}} -Passthru -Show None -Strict
    Test-ResultPester $Pester
  } catch {
     $InfoLogger.PSFatal("$_",$_.Exception)
     throw $_
  }
}

Next, I analyze the result :

function Test-ResultPester {
    #We are trying to find out if all the tests are successful otherwise
    #  there were exceptions in the tests that failed.

 param($Report)
    $isError=$isFailure=$false
    $isInconclusive=$Report.InconclusiveCount -ne 0
    foreach ($PesterTest in $Report.TestResult)
    {
        if (-not $isFailure)
        {  $isFailure=$PesterTest.Result -eq 'Failed'}

        if (-not $isError)
        { 
           $isError=$($PesterTest.Result -eq 'Failed') -And ($PesterTest.ErrorRecord.FullyQualifiedErrorId -ne 'PesterAssertionFailed')
        }
    }
    $isPassed =($isError -eq $false) -and ($isFailure -eq $false)
    $Result=[pscustomobject]@{
        PSTypeName='ReportPester';
        isError=$isError;
        isFailure=$isFailure;
        isInconclusive=$InconclusiveCount
        isPassed=$isPassed;
        isAllPassed=$isPassed -and ($isInconclusive -eq $false)
    } 
    if ($Result.isAllPassed -eq $false)
    {
        $Report.TestResult|
          Where-Object { $_.Result -eq 'Failed' }|
          ForEach-Object {
           if ($_.ErrorRecord.FullyQualifiedErrorId -eq 'PesterAssertionFailed')
           { $Infologger.PSError("$($_.Name) : $($_.FailureMessage)") }
           else
           { $InfoLogger.PSFatal("$($_.Name) : $($_.FailureMessage)",$_.ErrorRecord.Exception) } 
          }
    }   
    if ($Result.isError) 
    {throw "The function '$((Get-PSCallStack)[1].Command)' raises an exception."} 
    Return $Result
}

The original result of pester does not contain the information 'isError', what is normal considering the original use of Pester.

[LaurentDardenne]

Just for information, maybe this project PSRule seems more appropriate.