search

pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

v5: Discussion #1218

Closed nohwnd closed 3 years ago

nohwnd commented 5 years ago

🙋‍♀️

I am raising a few questions in v5 readme and this is the place to start discussion so you don't have to go through all the respective issues and ask there.

bergmeister commented 4 years ago

ScriptAnalyzer cannot know how a variable is being used by Pester due to it being a DSL. ScriptAnalyzer is a linter and the nature of linters is that there can be false positives that the tools cannot make a decision on and need human input. Therefore in that case, that warning should be suppressed specifically for that specific variable in that file. Suppressing is a way of stating in the source code that 'I have reviewed the warning and consider it to be not an issue'.

nohwnd commented 4 years ago

@bergmeister that sounds very annoying to do manually. Does PSSA consider all scriptblocks to form their own scope?

Could PSSA consider all Before and After blocks to be dot-sourced, and therefore accessible in the current and all child scopes?

bergmeister commented 4 years ago

@nohwnd At the moment, PSSA's variable and parameter analysis is limited to a per-scriptblock basis, even Begin/Process blocks do not see each other. This is due to PowerShell's dynamic nature, where it cannot even assume that a child scriptblock is invoked in the current scope due to PowerShell's dynamic nature, see https://github.com/PowerShell/PowerShell/issues/12287 (allow-listing certain known cmdlets that do like e.g. Where-Object is considered though.). Therefore making PSSA aware of Pester's DSL has a long way to go but we'd for sure consider PRs. Sure, it can be done, but there's loads of things that could be done (like e.g. the Begin/Process issue) and therefore I want to quote the comment of @rjmholt here again as it's also a case of adapting the user's mindset (as there is simply not enough time from the PowerShell team or the community to fulfil all those asks):

Ultimately this rule is supposed to be a helpful heuristic rather than an absolute.

nohwnd commented 4 years ago

I totally understand all the arguments, here and in the linked comments. I am not looking for a perfect solution that will catch every case. Rather I am looking for a solution that will be mostly correct, requiring minimal manual intervention in the most used scenarios.

rjmholt commented 4 years ago

It would be possible to consider them dot-sourced, but if the Before* and After* blocks are provided out of order, the underlying assumption wouldn't play out.

I actually played around with ingesting Pester ASTs and handling them specially. It's quite possible (if you or I can read a Pester test and understand it without executing it, so can PSSA), but would be hardcoded to Pester's current layout and be tricky to change.

It's still something I'm interested in doing, just because it would be nice for the linter to encourage people to use good tools like Pester, but I put off the work to invest in PSSA2, and possibly come up with more general solutions, at least for the dot-sourcing. Essentially a good solution would consist of:

To be perfectly clear, I don't personally consider many of the outstanding issues in the declared vars rule to be bugs, but rather feature requests to improve the heuristics. In Pester's case, the heuristic depends on an exact understanding of Pester's semantics and whether any of that is likely to change.

nohwnd commented 4 years ago

To be perfectly clear, I don't personally consider many of the outstanding issues in the declared vars rule to be bugs, but rather feature requests to improve the heuristics.

👍👍👍

BoSorensen commented 4 years ago

Although being told to ignore certain PSScriptAnalyzer warnings is a slippery slope to ignoring warnings in general. But any minor improvement would be very welcome.

AlexRedkin commented 4 years ago

Good day! I've got a question regarding mocking module functions. I import module and try to mock it's function, but it does not work. Could somebody please tell me what am I doing wrong? Here is the .psm1 file:

using module psPAS

class Sample {
    Sample() {
    }

    [void] login([PSCredential] $credential) {
        New-PASSession -Credential $credential -BaseURI "https://none"
    }
}

function Get-Sample(){
    $sample = [Sample]::new()
    return $sample
}

Export-ModuleMember -Function Get-Sample

And here is the script I am trying to execute.

Import-Module "C:\Users\Admin\Downloads\class.psm1"

Import-Module -Name Pester -RequiredVersion 5.0.2

Describe "Sometest"{
    Context "testContext"{

        Mock -ModuleName class -CommandName Get-Sample -MockWith {return "bla-bla"}

        It "test"
            get-sample | should -Be "bla-bla"
        }

    }

I've already read documentation and tried to reproduse solutions from here https://github.com/pester/Pester/issues/1351

nohwnd commented 4 years ago

@AlexRedkin That Mock should be inside of BeforeAll block.

AlexRedkin commented 4 years ago

@AlexRedkin That Mock should be inside of BeforeAll block.

Thank you for the clarification and awesome product! It works now!

deadlydog commented 4 years ago

For anybody else that upgraded their machine to Pester 5 (on purpose, or accidentally with a sweeping Update-Module command like me 😛 ) and doesn't have time to change their test code to account for the breaking changes, you can still use Pester v4 by sticking code similar to this at the top of your test scripts:

Install-Module -Name Pester -RequiredVersion 4.10.1 -Repository PSGallery
Import-Module -Name Pester -RequiredVersion 4.10.1

@nohwnd You could maybe make a note of this in the Pester ReadMe (perhaps in the Questions section) for others to be able to get back up and running quickly.

petercharleston commented 4 years ago

Hi everyone, I am getting the same error when using pester runner in a pipeline task I tried installing/importing version 4 module but get same error

The code i am using:

`[CmdletBinding()] param( [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName )

Install-Module -Name Pester -RequiredVersion 4.6.0 -Repository PSGallery Import-Module -Name Pester -RequiredVersion 4.6.0

requires -Module Az.Accounts,Az.Resources

describe 'Template validation' { it 'template passes validation check' { $parameters = @{ TemplateFile = 'server.json' ResourceGroupName = $ResourceGroupName adminUsername = 'adam' adminPassword = (ConvertTo-SecureString -String 'testing' -AsPlainText -Force) vmName = 'TESTING' } (Test-AzResourceGroupDeployment @parameters).Details | should -Benullorempty } }`

nohwnd commented 4 years ago

What is the error?

petercharleston commented 4 years ago

WARNING: You are using Legacy parameter set that adapts Pester 5 syntax to Pester 4 syntax. This parameter set is deprecated, and does not work 100%. The -Strict and -PesterOption parameters are ignored, and providing advanced configuration to -Path (-Script), and -CodeCoverage via a hash table does not work. Please refer to https://github.com/pester/Pester/releases/tag/5.0.1#legacy-parameter-set for more information.

System.Management.Automation.RuntimeException: No test files were found and no scriptblocks were provided. at Invoke-Pester, /home/vsts/.local/share/powershell/Modules/Pester/5.0.2/Pester.psm1: line 4197 at , /home/vsts/work/_tasks/Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9/0.1.3/Pester.ps1: line 160

[error]Write-Error: Pester returned errors

[error]Write-Error: Pester returned errors

Pester Script finished Finishing: Pester

bergmeister commented 4 years ago

@petercharleston Are you sure you haven't already imported v5 accidentally before running your install & import script because your log indicates that you are invoking pester v5? Generally I suggest to rather use -MaximumVersion 4.99 on the Install-Module and Import-Module cmdlets.

deadlydog commented 4 years ago

Generally I suggest to rather use -MaximumVersion 4.99 on the Install-Module and Import-Module cmdlets.

I'd typically agree with this, but I've found there's issues with -MaximumVersion in PSGet v2 and it is often ignored or errs out. They're supposed to be fixed PSGet v3, but that's still in beta.

petercharleston commented 4 years ago

This is at the top of the pester task: Pester Test Runner Description : Run Pester tests by either installing the latest version of Pester at run time (if possible) or using the version shipped with the task (4.6.0) Version : 0.1.3 Author : Pester

Could you advise on where or how i could have imported v5 accidentally please?

This pipeline was working on March 19th but did not work from Jun 16th

how could i force the task to use

Here is my pester script server.template.tests.ps1: `[CmdletBinding()] param( [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string]$ResourceGroupName )

Install-Module -Name Pester -MaximumVersion 4.99 -Repository PSGallery Import-Module -Name Pester -MaximumVersion 4.99

requires -Module Az.Accounts,Az.Resources

describe 'Template validation' { it 'template passes validation check' { $parameters = @{ TemplateFile = 'server.json' ResourceGroupName = $ResourceGroupName adminUsername = 'adam' adminPassword = (ConvertTo-SecureString -String 'testing' -AsPlainText -Force) vmName = 'TESTING' } (Test-AzResourceGroupDeployment @parameters).Details | should -Benullorempty } }`

I tried updating the pester script to include -MaximumVersion 4.99 but get the same error

here is my task in steps:

johndog commented 4 years ago

In 5.0.2, $TestDrive seems to return $nullall the time now. Not listed as a breaking change.

(I upgraded from 4 to 5 unintentionally, looks like I need to go back)

plastikfan commented 4 years ago

Is it possible to mock out a scriptblock? I asked because scriptblocks are un-named functions and as far as I can see, Pester mocking depends on seeing the function invocation by it's name. I have a function that takes a scriptblock as a variable, but for the life of me, I've no idea how to mock out the script block and then test that that block has been invoked in a test.

Consider this function invoke-ForeachFile (invokes a custom script block for every file passed in via pipeline):

function invoke-ForeachFile {

  [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSShouldProcess', '')]
  [CmdletBinding(SupportsShouldProcess)]
  param(
    [Parameter(Mandatory, ValueFromPipeline = $true)]
    [System.IO.FileSystemInfo]$pipelineItem,

    [Parameter()]
    [scriptblock]$Condition = ( { return $true; }),

    [Parameter(Mandatory)]
    [scriptblock]$Body,

   # Some details omitted 
  )

  process {
    if (-not($broken)) {
      if (-not($pipelineItem.Attributes -band [System.IO.FileAttributes]::Directory)) {
        $pipelineFileInfo = [System.IO.FileInfo]$pipelineItem;

        if ($Condition.Invoke($pipelineFileInfo)) {
          $collection += $pipelineFileInfo;

          $shouldProcess = $PSCmdlet.ShouldProcess($pipelineFileInfo.FullName, $Description);
          $result = Invoke-Command -ScriptBlock $Body -ArgumentList @(
            $pipelineFileInfo, $index, $PassThru, $trigger, -not($shouldProcess)
          );

          # some details ommitted
        }
      }
    }
  }
}

... and the test:

  Context 'WhatIf not specified in custom scriptblock' {
    It 'given: files piped from same directory, WhatIf flagged' -Tag 'Current' {
      [int]$count = 0;
      [scriptblock]$block = { # THIS IS THE ScriptBlock, how to mock this? 
        [CmdletBinding(SupportsShouldProcess)]
        param(
          [System.IO.FileInfo]$FileInfo,
          [int]$Index,
          [System.Collections.Hashtable]$PassThru,
          [boolean]$Trigger
        )
        [ref]$count++; # THIS DOES NOT WORK (I'm not sure if Powershell forms a Closure here)
        Write-Host ">>> $($FileInfo.Name), WhatIf: $($WhatIf), Count: $([ref]$count)";
      }

      [string]$directoryPath = './Tests/Data/csv';
      Get-ChildItem $directoryPath | invoke-ForeachFile -body $block -Description "Operation blade" -WhatIf;
      $count | Should -Be 2;
    }
  }

As a workaround, I tried to use a $count variable inside the block, so that the test can test if the block was called, but even this doesn't work, even with the [ref] attribute!

So is this possible, if not is there a workaround?

nohwnd commented 4 years ago

@plastikfan this won't work because $count++ will create a new local variable called count and will assign the result of the parent count + 1 to it. So you will never see the result outside of that scope.

Same as if you'd do: 

$count = 0
& {
    $count++

    Get-Variable -Scope Local -Name Count
    Get-Variable -Scope 1 -Name Count
}
"count is: $count"

Name                           Value
----                           -----
count                          1    <- the local var
count                          0    <- the parent var
count is: 0

But you can have are reference to an object and add to a property on it, that will work correctly, becuase you modify the value on the object instead of creating a local variable: 

$c = @{ 
    Count = 0
}
& {
    $c.Count++
}

"count is: $($c.Count)"
It 'given: files piped from same directory, WhatIf flagged' -Tag 'Current' {
      $container = @{
          Count = 0
      }

      [scriptblock]$block = { # THIS IS THE ScriptBlock, how to mock this? 
        [CmdletBinding(SupportsShouldProcess)]
        param(
          [System.IO.FileInfo]$FileInfo,
          [int]$Index,
          [System.Collections.Hashtable]$PassThru,
          [boolean]$Trigger
        )
        $container.Count++; # THIS DOES NOT WORK (I'm not sure if Powershell forms a Closure here)
        Write-Host ">>> $($FileInfo.Name), WhatIf: $($WhatIf), Count: $($container.Count)";
      }

      [string]$directoryPath = 'c:\temp\';
      Get-ChildItem $directoryPath | invoke-ForeachFile -body $block -Description "Operation blade" -WhatIf;
      $container.Count | Should -Be 2;
    }
plastikfan commented 4 years ago

Thanks @nohwnd, I tried to use this technique (the [ref] attribute), because this is the way it has worked effectively for me in the past. The big difference that counts is the fact in my previous cases, the variable I was referencing was an object so it worked, in contrast with the way I was using it this time round on a scalar value, so thanks for clearing that up (I realise now that I can probably remove the [ref] attribute from my old code, it's probably redundant, but something tells me, it can't have worked, otherwise I wouldn't have put it in!). I assume, then that there is no resolution to being able to pass in a mock as a scriptblock. Just need to know so that I don't waste time trying to figure this out, I'll find another way like defining my own mock class/instance with embedded asserts, if Pester doesnt currently offer this capability.

thor commented 4 years ago

In 5.0.2, $TestDrive seems to return $nullall the time now. Not listed as a breaking change.

(I upgraded from 4 to 5 unintentionally, looks like I need to go back)

I do not experience the same here in Core, but I do experience that $TestDrive returns null in BeforeAll outside of a Describe block.

GitHed commented 4 years ago

I had this working in 4, but in 5, I can't see to use a variable in the describe block text:

This is basically a generic module checking script for module development that uses individual function files during dev, but gets packaged and signed as a single file during a build process.

Wrapping in BeforeAll doesn't allow the var to be used in the describe: Describe "Module Tests: $ModuleName" {.

BeforeAll {
    function Find-InParentPath {
        [CmdletBinding()]
        param(
            [String]
            $Path,

            [String]
            $Filter
        )

        Process {
            # Write-Verbose "$Path"
            $Items = Get-ChildItem -Path $Path -Filter $Filter
            if ($Items.Count -eq 1) {
                Write-Output (Get-Item -Path $Items.FullName)
            } else {
                Find-InParentPath -Path ((Get-Item -Path (Resolve-Path -Path $Path)).Parent.FullName) -Filter $Filter
            }
        }
    }

    $ManifestPath   = Find-InParentPath -Path $PSScriptRoot -Filter '*.psd1'
    $ModulePath     = $ManifestPath.Directory
    $ModuleName     = $ManifestPath.BaseName

    # Force reload the module from the file.
    Remove-Module -Force -Name $ModuleName
    $Manifest = Test-ModuleManifest -Path $ManifestPath
    Import-Module -Force -Name $ManifestPath

    # Did we load the right module?
    $Module = Get-Module -Name $ModuleName
    if (("$($Manifest.Name)" -ne "$($Module.Name)") -or ("$($Manifest.Name)" -ne "$($ModuleName)") -or ("$($Module.Name)" -ne "$($ModuleName)")) {
        Write-Warning "$($Manifest.Name) -ne $($Module.Name) -ne $($ModuleName)"
        throw "$($Manifest.Name) -ne $($Module.Name) -ne $ModuleName"
    }
}

Describe "Module Tests: $ModuleName" {
    Context 'Module Setup' {
        It "has the root module $ModuleName.psm1" {
            "$ModulePath\$ModuleName.psm1" | Should -Exist
        }

        It "has the a manifest file of $ModuleName.psd1" {
            "$ModulePath\$ModuleName.psd1" | Should -Exist
            "$ModulePath\$ModuleName.psd1" | Should -FileContentMatch "$ModuleName.psm1"
        }

        It "$ModuleName is valid PowerShell code" {
            $psFile = Get-Content -Path "$ModulePath\$ModuleName.psm1" -ErrorAction Stop
            $errors = $null
            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
            $errors.Count | Should -Be 0
        }

        It "Module's filelist files exist" {
            ForEach ($File in $Module.FileList) {
                $File | Should -Exist
            }
        }

        It "Module's filelist missing files" {
            $Files = Get-ChildItem -Recurse $ModulePath -File
            ForEach ($File in $Module.FileList) {
                $File | Should -BeIn $Files.FullName
            }
        }
    }
}
nohwnd commented 4 years ago

@GitHed What does not work in that example?

GitHed commented 4 years ago

@nohwnd

So, I was trying to simplify the issue to a specific case, but really, my idea of dynamic testing is broken.

I also probably should have just posted the full file instead of cutting parts to try to narrow the issue, I also had mistaken my first issue to just be a fault of my own.

I do seem to need to include a function in two locations now, in order to get the variable available for the describe context.

The current issue is I create context blocks inside of a function, and those don't seem to run because they aren't discovered.

Here's the full test file:


BeforeAll {
    function Find-InParentPath {
        [CmdletBinding()]
        param(
            [String]
            $Path,

            [String]
            $Filter
        )

        Process {
            # Write-Verbose "$Path"
            $Items = Get-ChildItem -Path $Path -Filter $Filter
            if ($Items.Count -eq 1) {
                Write-Output (Get-Item -Path $Items.FullName)
            } else {
                Find-InParentPath -Path ((Get-Item -Path (Resolve-Path -Path $Path)).Parent.FullName) -Filter $Filter
            }
        }
    }

    $ManifestPath   = Find-InParentPath -Path $PSScriptRoot -Filter '*.psd1'
    $ModulePath     = $ManifestPath.Directory
    $ModuleName     = $ManifestPath.BaseName

    Write-Verbose "ManifestPath: $($ManifestPath)"
    Write-Verbose "ModulePath:   $($ModulePath)"
    Write-Verbose "ModuleName:   $($ModuleName)"

    # Force reload the module from the file.
    Remove-Module -Force -Name $ModuleName
    $Manifest = Test-ModuleManifest -Path $ManifestPath
    Import-Module -Force -Name $ManifestPath

    # Did we load the right module?
    $Module = Get-Module -Name $ModuleName
    if (("$($Manifest.Name)" -ne "$($Module.Name)") -or ("$($Manifest.Name)" -ne "$($ModuleName)") -or ("$($Module.Name)" -ne "$($ModuleName)")) {
        Write-Warning "$($Manifest.Name) -ne $($Module.Name) -ne $($ModuleName)"
        throw "$($Manifest.Name) -ne $($Module.Name) -ne $ModuleName"
    }
}

Describe "Module Tests: $ModuleName" {
    BeforeAll {
        function Find-InParentPath {
            [CmdletBinding()]
            param(
                [String]
                $Path,

                [String]
                $Filter
            )

            Process {
                # Write-Verbose "$Path"
                $Items = Get-ChildItem -Path $Path -Filter $Filter
                if ($Items.Count -eq 1) {
                    Write-Output (Get-Item -Path $Items.FullName)
                } else {
                    Find-InParentPath -Path ((Get-Item -Path (Resolve-Path -Path $Path)).Parent.FullName) -Filter $Filter
                }
            }
        }

        function Test-Function {
            [CmdletBinding()]
            Param (
                [String]
                $Function,

                [String]
                $ModuleName,

                [ValidateScript({Test-Path -PathType Container -Path $_})]
                [String]
                $ModulePath,

                [ValidateScript({Test-Path -PathType Container -Path $_})]
                [String]
                $TestsPath,

                [ValidateSet('Private', 'Public')]
                [String]
                $Context,

                [Switch]
                $CheckFile
            )

            Context "Function: $ModuleName\$Function" {
                if ($CheckFile) {
                    It "$Function.ps1 should exist" {
                        "$ModulePath\$Context\$Function.ps1" | Should -Exist
                    }

                    if (Test-Path -Path "$ModulePath\$Context\$Function.ps1") {
                        It "$Function.ps1 is valid PowerShell code" {
                            $psFile = Get-Content -Path "$ModulePath\$Context\$Function.ps1" -ErrorAction Stop
                            $errors = $null
                            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
                            $errors.Count | Should -Be 0
                        }

                        It "$Function.ps1 should be an advanced Function" {
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'Function'
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'cmdletbinding'
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'param'
                        }

                        It "$Function.ps1 should contain Write-Verbose blocks" {
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'Write-Verbose'
                        }
                    }
                }

                Context "Get-Help $ModuleName\$Function" -Tag 'Help' {
                    $Help = Get-Help -Name $ModuleName\$Function -Category Function -ErrorAction SilentlyContinue

                    It "$Function should have a Synopsis" {
                        $Help.SYNOPSIS | Should -BeOfType String
                        $Help.SYNOPSIS | Should -Not -BeNullOrEmpty
                    }

                    It "$Function should have a Description" {
                        $Help.Description.Text | Should -BeOfType String
                        $Help.Description.Text | Should -Not -BeNullOrEmpty
                    }

                    It "$Function should have at least one Example" {
                        ForEach ($Example in $Help.Examples) {
                            $Example.example.code | Should -Not -BeNullOrEmpty
                        }
                    }

                    Context "Get-Help $Function -Parameter *" {
                        ForEach ($HelpParam in $Help.Parameters.Parameter) {
                            if (('WhatIf' -ne $HelpParam.Name) -and ('Confirm' -ne $HelpParam.Name)) {
                                It "Parameter $($HelpParam.Name) should have a Description" {
                                    $HelpParam.Description.Text | Should -BeOfType String
                                    $HelpParam.Description.Text | Should -Not -BeNullOrEmpty
                                }

                                if (('false' -eq $HelpParam.Required) -and ('switch' -ne $HelpParam.Type.Name)) {
                                    It "Parameter $($HelpParam.Name) should have a Default value if not Required" {
                                        $HelpParam.defaultValue | Should -Not -BeNullOrEmpty
                                    }
                                }
                            }
                        }
                    }
                }

                if ($CheckFile) {
                    Context "Pester Tests $Function" {
                        It ".\Tests\$Function.Tests.ps1 should exist" {
                            "$TestsPath\$Function.Tests.ps1" | Should -Exist
                        }

                        It ".\Tests\$Function.Tests.ps1 is valid PowerShell code" {
                            $psFile = Get-Content -Path "$TestsPath\$Function.Tests.ps1" -ErrorAction Stop
                            $errors = $null
                            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
                            $errors.Count | Should -Be 0
                        }
                    }
                }
            }
        }

        $ManifestPath   = Find-InParentPath -Path $PSScriptRoot -Filter '*.psd1'
        $ModulePath     = $ManifestPath.Directory
        $ModuleName     = $ManifestPath.BaseName
        $Module         = Get-Module -Name $ModuleName
        $TestsPath      = Find-InParentPath -Path $PSScriptRoot -Filter 'Tests'
        $PublicFiles    = Test-Path -Path "$($ModulePath)\Public" -PathType Container
        $PrivateFiles   = Test-Path -Path "$($ModulePath)\Private" -PathType Container

        Write-Verbose "ManifestPath: $($ManifestPath)"
        Write-Verbose "ModulePath:   $($ModulePath)"
        Write-Verbose "ModuleName:   $($ModuleName)"
        Write-Verbose "TestsPath:    $($TestsPath)"

        # Get the module functions
        $ModuleFunctions_Public  = Get-Command -Module $ModuleName -CommandType Function
        $ModuleFunctions_All     = $Module.Invoke({Get-Command -CommandType Function -Module $ModuleName})
        $ModuleFunctions_All     = $ModuleFunctions_All | Where-Object {$_.ModuleName -eq $ModuleName}
        $ModuleFunctions_Private = Compare-Object -ReferenceObject $ModuleFunctions_All -DifferenceObject $ModuleFunctions_Public | Select-Object -ExpandProperty InputObject

        Write-Verbose "ModuleFunctions_All.Count: $($ModuleFunctions_All.Count)"
        Write-Verbose "ModuleFunctions_Public.Count: $($ModuleFunctions_Public.Count)"
        Write-Verbose "ModuleFunctions_Private.Count: $($ModuleFunctions_Private.Count)"

        Write-Verbose "ModuleFunctions_Private: $($ModuleFunctions_Private)"
        Write-Verbose "ModuleFunctions_Public:  $($ModuleFunctions_Public)"
    }

    Context 'Module Setup' {
        It "has the root module $ModuleName.psm1" {
            "$ModulePath\$ModuleName.psm1" | Should -Exist
        }

        It "has the a manifest file of $ModuleName.psd1" {
            "$ModulePath\$ModuleName.psd1" | Should -Exist
            "$ModulePath\$ModuleName.psd1" | Should -FileContentMatch "$ModuleName.psm1"
        }

        It "$ModuleName is valid PowerShell code" {
            $psFile = Get-Content -Path "$ModulePath\$ModuleName.psm1" -ErrorAction Stop
            $errors = $null
            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
            $errors.Count | Should -Be 0
        }

        It "Module's filelist files exist" {
            ForEach ($File in $Module.FileList) {
                $File | Should -Exist
            }
        }

        It "Module's filelist missing files" {
            $Files = Get-ChildItem -Recurse $ModulePath -File
            ForEach ($File in $Module.FileList) {
                $File | Should -BeIn $Files.FullName
            }
        }
    }

    Context 'Functions' {
        if ($null -ne $ModuleFunctions_Private) {
            if (Get-Member -Name 'Count' -InputObject $ModuleFunctions_Private) {
                if ($ModuleFunctions_Private.Count -gt 0) {
                    Context 'Private Functions' -Tag 'Private' {
                        ForEach ($Function in $ModuleFunctions_Private) {
                            Test-Function -Function $Function -ModuleName $ModuleName -ModulePath $ModulePath -TestsPath $TestsPath -Context 'Private' -CheckFile:$PrivateFiles
                        }
                    }
                }
            }
        }

        if ($null -ne $ModuleFunctions_Public) {
            if (Get-Member -Name 'Count' -InputObject $ModuleFunctions_Public) {
                Context 'Public Functions' -Tag 'Public' {
                    ForEach ($Function in $ModuleFunctions_Public) {
                        Test-Function -Function $Function -ModuleName $ModuleName -ModulePath $ModulePath -TestsPath $TestsPath -Context 'Public' -CheckFile:$PublicFiles
                    }
                }
            }
        }
    }
}

And it doesn't matter if it's combined or the single file, the Functions context will not have any discoverable tests, and thus just won't run.

And for reference, here's the difference in test numbers between 4 and 5.

5.0.2

Starting discovery in 1 files.
Discovery finished in 48ms.
Tests completed in 1.12s
Tests Passed: 5, Failed: 0, Skipped: 0 NotRun: 0

4.9.0

    ____            __
   / __ \___  _____/ /____  _____
  / /_/ / _ \/ ___/ __/ _ \/ ___/
 / ____/  __(__  ) /_/  __/ /
/_/    \___/____/\__/\___/_/
Pester v4.9.0
Executing all tests in 'F:\Projects\Modules\ModuleName\Tests\ModuleName.Tests.ps1'
Tests completed in 24.43s
Tests Passed: 225, Failed: 52, Skipped: 0, Pending: 0, Inconclusive: 0
nohwnd commented 4 years ago

@GitHed Your Test-Function is defined in BeforeAll which will make it run after Discovery, so when you reach the Functions Context it won't generate anything because the Test-Function is not defined yet. Also some of your variables won't be defined in your tests like $Help won't be defined in your tests, because you define them in the body of Describe / Context.

GitHed commented 4 years ago

@nohwnd Thanks for the response!

So I re-arranged that code because reading suggested it needed to be in a beforeall block to work. But I've basically gone back to what I had before...

Side tracking here a little: It does seem that there's an empty context variable in the it blocks: Get-Content : Cannot find path 'F:\Projects\Modules\ModuleName\@{Name=; Path=}\.ps1' because it does not exist.

I pass a string (which granted is not there), but not an object with properties. 

$PesterResults.Passed[25]

Name              : Context leaks?
Path              : {Module Tests: ModuleName, Functions, Public Functions, Function: ModuleName\FunctionName...}
Data              : {}
ExpandedName      : Context leaks?
Result            : Passed
ErrorRecord       : {}
StandardOutput    :
Duration          : 00:00:00.0203784
ItemType          : Test
Id                :
ScriptBlock       :
                                $Context.name | Should -BeNullOrEmpty
                                $Context.path | Should -BeNullOrEmpty

Tag               :
Focus             : False
Skip              : False
Block             : [-] Function: ModuleName\FunctionName
First             : True
Last              : False
Include           : False
Exclude           : False
Explicit          : False
ShouldRun         : True
Executed          : True
ExecutedAt        : 7/29/2020 3:35:54 AM
Passed            : True
Skipped           : False
UserDuration      : 00:00:00.0083888
FrameworkDuration : 00:00:00.0119896
PluginData        :
FrameworkData     :

Changing the parameter name of the test-function function stops outputting the empty context object in the error.

[-] Module Tests: ModuleName.Functions.Public Functions.Function: ModuleName\FunctionName.FunctionName.ps1 should exist 12ms (7ms|5ms)
 Expected path 'F:\Projects\Modules\ModuleName\\.ps1' to exist, but it did not exist.

You can check this with this simple test file:

Describe "describe" {
    Context 'Context' {
        it 'it' {
            $Context | Should -Not -BeNullOrEmpty
        }
    }
}

Now, how do I actually get a variable into an it block? I noticed none of the examples really show that, but they do suggest this should work:

Context "Function: $ModuleName\$Function" {
        BeforeAll {
            function Get-FunctionFilePath {
                Write-Output "$($ModulePath)\$($Type)\$($Function).ps1"
            }
        }

        It "$Function.ps1 should exist" {
            $FilePath = Get-FunctionFilePath
            $FilePath | Should -Exist
        }

Defining Get-FunctionFilePath in a Begin section or before the Context block of the Test-Function function just results in not defined in the it blocks, placing it in BeforeAll makes it available, but still null.

plastikfan commented 4 years ago

Using the & operator to invoke a function in a Pester test is proving to be difficult. The following code DOES work:

  BeforeAll {
    function Show-Greeting {
      param(
        [string]$Person
      )
      Write-Host "Hello $Person";  
    }

  Context 'given: a test function' {
    It 'should: be invokable via the & operator' {
      $parameters = @{
        'Person' = 'The Nephilim'
      }
      & Show-Greeting @parameters;
    }
  }

The invoke is directly inside the test case. However, a module function that uses & operator to invoke a function indirectly does not work. The only way that this works, is when the function being invoked is moduule source code, not a test function defined in the BeforeAll or It block. So this does not work:

      It 'Should: traverse creating directories only' -Tag 'Current' {

        function Show-InternalMirror {
          param(
            [Parameter(Mandatory)]
            [System.IO.DirectoryInfo]$Underscore,

            [Parameter(Mandatory)]
            [int]$Index,

            [Parameter(Mandatory)]
            [System.Collections.Hashtable]$PassThru,

            [Parameter(Mandatory)]
            [boolean]$Trigger,

            [Parameter(Mandatory)]
            [string]$Format
          )

          [string]$result = $Format -f ($Underscore.Name);
          Write-Host "Custom function; Show-InternalMirror: '$result'";
          @{ Product = $Underscore }
        }
        [System.Collections.Hashtable]$parameters = @{
          'Format' = '---- {0} ----'
        }
        Invoke-MirrorDirectoryTree -Path $sourcePath `
          -DestinationPath $destinationPath -CreateDirs `
          -Functee 'Show-InternalMirror' -FuncteeParams $parameters -WhatIf:$whatIf;
      }

So Invoke-MirrorDirectoryTree is a module function that invokes Functee with the & operator. The only (very un-desirable) workaround is to include the test function Show-InternalMirror within the module, but I don't like doing this because it is a function that is of no benefit to the end-user and is not required in a production environment. I have a lot of testcases that are afflicted with this issue, which means I have a lot of duff test functions in the production module.

The error shown is something like this

     | Invoke-TraverseDirectory(top-level) Error: (Exception calling "Invoke" with "4" argument(s): "The term
     | 'Show-InternalMirror' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
     | the spelling of the name, or if a path was included, verify that the path is correct and try again."), for item:
     | 'traverse'

Is there an issue with using & operator for internally defined Pester test functions and how can I define a test function within a Pester test case that can be invoked by a module function under test?

EDIT: I tried InModuleScope and that makes no difference.

plastikfan commented 4 years ago

Are there any guidelines for writing Pester tests that run on local machine but also work on Azure? At the moment, you can write tests thast work locally, but they don't run on Azure, there are quite a few differences. I'm finding that if I tweek my build process (I'm using Invoke-Build) to try and get tests running on Azure, they will break the regular local run. Problems are mainly to do with importing the built module and getting access to the exported functions that don't seem to be available when running the tests on Azure.

hugh-martin commented 4 years ago

I realize this is probably more of a PowerShell issue than a Pester issue, but are there any workarounds for testing functions that return PowerShell v5 class objects? When I debug a Pester test, I can see the object, but the test itself sees it as null.

nohwnd commented 4 years ago

@plastikfan Show-InternalMirror question: Did that work in Pester 4? To me it seems like it should never work, or work for a very specific case, but not for many others.

@plastikfan Azure question: Do you have an example of pipeline that does not work, while it works locally (or the other way around)? Is this just Pester v5 problem?

nohwnd commented 4 years ago

@hugh-martin could you show an example please? The class support in Pester is quite limited, though.

plastikfan commented 4 years ago

@plastikfan Show-InternalMirror question: Did that work in Pester 4? To me it seems like it should never work, or work for a very specific case, but not for many others.

Hi @nownd, as I'm relatively new to Pester, I am not sure if this would work in v4. I've just come to accept this is a current limitation of Pester. Using the & op on a named function simply does not work unfortunately, so at the moment as a workaround I have to define dummy test functions that get delivered with the module (which is less than desirable).

@plastikfan Azure question: Do you have an example of pipeline that does not work, while it works locally (or the other way around)? Is this just Pester v5 problem?

And this is not an issue with Pester, it is purely an Azure issue and what makes this worse is that there is next to no information about testing Powershell modules in an Azure pipeline. There is documentation on using powershell scripting, but modules have not been adequately catered for. The main issue is the scope of a powershell session within a task/job/stage etc, because what you'll find is you can define a Job to bootstrap the session (install dependencies), but the next job will fail, becasue there is a new session and what what bootstrapped no longer exits. I was hoping somebody else here might have had better luck testing PowerrShell modules on Azure. I'm attending a virtual tech meeting with MS talking about Azure pipelines. I'm going to bring the issue of PS modules on Azure and ask why there's no documetation on this.

nohwnd commented 4 years ago

@plastikfan you can pass a function reference which will work, and probably will also still work nicely with the name. Because & will invoke both name and well as the commnd info object. -Functee (Get-Command Show-InternalMirror), and [object] $Functee

nohwnd commented 4 years ago

@plastikfan I run all Pester tests in v5.0 in AzDO, but my way might be specific.

plastikfan commented 4 years ago

@plastikfan I run all Pester tests in v5.0 in AzDO, but my way might be specific.

Ok, is the pipeline public? I'd like to take a look to see if I can resolve my issues.

plastikfan commented 4 years ago

@plastikfan you can pass a function reference which will work, and probably will also still work nicely with the name. Because & will invoke both name and well as the commnd info object. -Functee (Get-Command Show-InternalMirror), and [object] $Functee

I didnt know there was such a thing as a function reference (via Get-Command). I can see that will resolve another issue that I have developed a work-around for, which having seen this is now redundant. I'll try this technique out in my test and hopefully that will work.

nohwnd commented 4 years ago

@plastikfan yes it is public: https://nohwnd.visualstudio.com/Pester/_build?definitionId=6&_a=summary

plastikfan commented 4 years ago

Great thanks @nohwnd I'll study that in great detail and see where I went wrong. Cheers

hugh-martin commented 4 years ago

@nohwnd It looks like I had the same problem as @AlexRedkin. I had to move my mock into a BeforeAll or it block. Sorry for the false alarm.

hugh-martin commented 4 years ago

Am I correct regarding BeforeAll and AfterAll blocks:

nohwnd commented 4 years ago

@hugh-martin

hugh-martin commented 4 years ago

Thanks @nohwnd.

hugh-martin commented 4 years ago

I'm trying to make sure I understand how scoping works, but my test script is getting some unexpected results.

BeforeAll {
    function GoForIt {
        Write-Output "bob"
    }
}

Describe "Function test" {

    Context "Test 1" {

        it "'bob' (hard-coded) should be bob" {
            $myvar = GoForIt
            $myvar | Should -Be "bob"
        }

        it "There should be a myvar variable (1)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1 -Because "the variable was just defined and used"
        }
    }

    Context "Test 2" {

        it "There should NOT be a myvar variable (1)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 0 -Because "this is a new Context block"
        }

    }

    Context "Test 3" {

        BeforeAll {
            $myvar = GoForIt
        }

        it "There should be a myvar variable (1)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1 -Because "The variable was defined in the BeforeAll block"
        }

        it "'$myvar' (variable) should be bob" {
            $myvar | Should -Be "bob"
        }

        it "There should be a myvar variable (2)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1
        }
    }
}

Here are the results:

Describing Function test
 Context Test 1
   [+] 'bob' (hard-coded) should be bob 10ms (7ms|2ms)
   [-] There should be a myvar variable (1) 14ms (12ms|2ms)
    Expected 1, because the variable was just defined and used, but got 0.
    at (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1 -Because "the variable was just defined and used", C:\MyTemp\Test1.Tests.ps1:17
    at <ScriptBlock>, C:\MyTemp\Test1.Tests.ps1:17
 Context Test 2
   [+] There should NOT be a myvar variable (1) 8ms (6ms|2ms)
 Context Test 3
   [+] There should be a myvar variable (1) 8ms (6ms|2ms)
   [+] '' (variable) should be bob 9ms (7ms|2ms)
   [+] There should be a myvar variable (2) 7ms (5ms|2ms)
Tests completed in 765ms
Tests Passed: 5, Failed: 1, Skipped: 0 NotRun: 0
nohwnd commented 4 years ago

@hugh-martin

hugh-martin commented 4 years ago

So when you say

BeforeEach, It and AfterEach now run in the same scope, but are still isolated from their parent to avoid leaking variables and test cross-pollution,

it doesn't mean that all It blocks in a single parent (Context or Describe) block share the same scope as I thought. Does this mean that v5 is much more tightly scoped than v4? I apologize for the newbie questions; just trying to get the basics down. Pester has been great for operational validation, but I'm just starting to use it for code testing. Understanding scope means knowing how to structure the tests (or at least knowing how NOT to!). Thanks for the help.

hugh-martin commented 4 years ago

I rewatched your Pester 5 video and I think I understand how to restructure my code and code blocks to get the benefits of Pester 5.

nohwnd commented 4 years ago

@hugh-martin No problem. Yes in some ways the scoping is more tight and logical.

This is how it looks like in v5: 

# & { } is a new scope
# . { } is dot-sourcing into the current scope
# Each variable represents the scriptblock provided, to the 
# Pester block of that name, e.g. BeforeAll { $a = 10 }, 
# then in this diagram $BeforeAll = { $a = 10 }
# or It "is 10" { $a | Should -Be 10 }, 
# then $It = { $a | Should -Be 10 }

& {
    . $BeforeAll
    & {
        . $BeforeEach
        . $It
        . $AfterEach
    }
    . $AfterAll
}

And this is how it looked in v4. 

. $BeforeAll
& {
    . $BeforeEach
    &{
        . $It
    }
    . $AfterEach
}
. $AfterAll

Notice that v5 has one extra scope that isolates each group (that would be a top-level Describe block) from the script itself. So variables don't leak from first Describe in the file to the next one.

And that It is in the same scope as BeforeEach and AfterEach, this is to ensure that you can move a piece of your test into common setup and get the same behavior, and that you can use values that you set in It in your teardown (e.g. when making sure a file is deleted.

For completeness here is how it looks if we include ensuring that After* blocks will run, try catch or try finally does not create a new scope so it does not have any effect on our discussion, but I already wrote it so maybe you might find it useful: 

& {
    try{
        . $BeforeAll
        & {
            try {
                . $BeforeEach
                . $It
            }
            finally {
                . $AfterEach
            }
        }
    }
    finally {
        . $AfterAll
    }
}
hugh-martin commented 4 years ago

This is super helpful. Thank you.

nohwnd commented 4 years ago

Comment 100 💯