Clarification on Simple vs Comprehensive

[timker]

...it kind of looks like unit tests vs intergration tests

but the comment here looks like they are different https://github.com/PowerShell/Operation-Validation-Framework/issues/2

what is your definition of Simple/Comprehensive Tests?

[My-Random-Thoughts]

We way we use them is

Simple:
    Checks to make sure all test requirements are installed/available locally
    Connection test to remote server to be checked to make sure it exists

Comprehensive:
    All the tests we want to perform

This way we have a quick fail if something is missing that should not be.

[ArieHein]

Would you have the same tests in Simple added into Comprehensive as well ?

[davidwallis]

No.. I do this:

    if ($PSCmdlet.ParameterSetName -eq "ComputerName") {
        Write-Verbose "Establishing session with: $($ComputerName)"
        $Session = new-pssession -Computer $ComputerName -Credential $credential -UseSSL:$UseSSL.IsPresent -ErrorAction Stop
    }

    $simpleTests = Get-OperationValidation `
        -ModuleName 'ServerBuildValidation' `
        -TestType Simple `
        -Verbose:($PSCmdlet.MyInvocation.BoundParameters["Verbose"].IsPresent -eq $true)

    if ($null -eq $simpleTests) {
        Write-error "No tests found" -ErrorAction stop
    }

    $simpleOvfParams        = @{ ComputerName = $ComputerName; Session = $Session; UseSSL = $UseSSL.IsPresent }
    $comprehensiveOvfParams = @{ ComputerName = $ComputerName; Session = $Session; OctopusApiKey = $OctopusApiKey }

    try {
        $simpleResults = $simpleTests | Invoke-OperationValidation -Overrides $simpleOvfParams
        if (($simpleResults | Where-Object -Property Result -EQ -Value 'Failed' | measure-object).Count -gt 0) {
            Write-Error "Simple tests failed, Stopping." -ErrorAction stop
        }

        $comprehensiveTests = Get-OperationValidation -ModuleName 'ServerBuildValidation' -TestType Comprehensive -Verbose
        $comprehensiveResults = $comprehensiveTests | Invoke-OperationValidation -Overrides $comprehensiveOvfParams
    }
    catch [System.Exception] {
        Write-Error "Exception occurred in $($MyInvocation.MyCommand): $($_.Exception.Message)"
        Write-Error  ($_.Exception | Format-List -Force | Out-String)
    }

    $resultTypes = $comprehensiveResults | Group-Object -Property Result

    $summary = @{
        Passed       = ($resultTypes | Where-Object Name -eq 'Passed'      ).Count
        Failed       = ($resultTypes | Where-Object Name -eq 'Failed'      ).Count
        Skipped      = ($resultTypes | Where-Object Name -eq 'Skipped'     ).Count
        Inconclusive = ($resultTypes | Where-Object Name -eq 'Inconclusive').Count
        Pending      = ($resultTypes | Where-Object Name -eq 'Pending'     ).Count
        Total        = ($resultTypes | Measure-Object -Property Count -Sum ).Sum
    }

    if ($Detailed.IsPresent -eq $false) { return $summary }

    return (Format-AsHtml `
        -ComputerName $ComputerName `
        -Summary $summary `
        -ComprehensiveResults $comprehensiveResults `
        -OperatingSystem (Invoke-Command -Session $Session -ScriptBlock { Get-CimInstance -Class Win32_OperatingSystem }).Caption `
        -BuildVersion    (Invoke-Command -Session $Session -ScriptBlock { Get-ItemProperty -Path 'HKLM:\System\Infrastructure' -ErrorAction SilentlyContinue }).'Build Script Version'
    )

edit: just noticed the passing of verbose through could be done better - also I've stripped out the top of the function to hide company specific info.. as I think its only the logic you need :) let me know if you want the full version and ill edit and gist it.

[davidwallis]

I then use the simple tests to check connectivity - IE all my tests are executed locally but test a remote machine via a lot of Invoke-Command

so simple tests are:

Does Test-WSMan work to the target machine being tested? Are the required dependencies on the client machine (IE the DLL's, modules, tools etc all present)

Comprehensive then actually runs the detailed testing that the machine is built as we expect.. If we hit any kind of bug or issue we then write a test so we never have that issue again.

At the moment its not ideal as people keep putting logic in tests and it shouldnt be, they should be helper functions within dot sourced scripts than can have their own unit tests with them.. IMHO.

The above helps be only run comprehensive if the simple tests fail and also get a summary that I use to build a HTML report to chuck into the automated emails.

[ArieHein]

Yes I assume for VM you just run both sets of test when you want comprehensive so you dont repeat the tests being coded twice. I mostly use to test my azure resources after they were created via terraform, so for example, in WebApp simple tests include, making sure the resource exists based on its name, that it is Enabled and in Running State and a simple http check to see it returns 200OK. My comprehensive test do all that simple does plus checking the location is in the desired location, that its connected to the right service plan, and additional http cheeks for health and apiversion verification.

Probably should use your method to incorporate both tests when comprehensive is used so i dont have to manage duplicate code.

[davidwallis]

I simply have that code in a function within the OVF module called Invoke-Validation, you could put a switch param called [switch]SimpleOnly to exit at that point