search

pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

Invoke-Pester does not respect tags when pester is loaded from a path #1065

Closed JamesCrompton-MSFT closed 6 years ago

JamesCrompton-MSFT commented 6 years ago

1. Provide a general summary of the issue in the Title above

We want our tests to load pester from a path so that we load a specific version from our branch and not a random version people may have on their machines. This scenario is blocked however because tags are no longer respected when pester is loaded from a path.

2. Describe Your Environment

Operating System, Pester version and PowerShell version: Pester version : 4.3.1 C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1\Pester.psd1 PowerShell version : 5.0.10586.117 OS version : Microsoft Windows NT 6.3.9600.0

3. Expected Behavior

Calling invoke-pester with a tag (or exclude tag) should respect that tag and only run the tests which match the filter. eg: invoke-pester -tag Unit -Script .\Test\SimpleRepro.Tests.ps1 should only run Unit tests or eg: invoke-pester -ExcludeTag DevEnvironment -Script .\Test\SimpleRepro.Tests.ps1 should not run DevEnvironment tests.

4.Current Behavior

All tests are run regardless of their tag

Repro steps: 1) Browse to C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1 and make a copy of pester into some other path (in my case "C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1.Copy" ) 2) Create a simple test which loads pester from a path rather than by name.

Import-Module "C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1.Copy\Pester.psd1"

Describe "pester tests." -Tag DevEnvironment { Context "Sanity test." { It "1 should be 1" ` { 1 | should be 1 } } } 3) Run Invoke-Pester against this file excluding the tag so the test should not be run. Result: When pester is loaded as a path the tag is ignored and the test is run anyway, if you load pester by name in the test "Ipmo Pester" then it works. The pester code is identical since it was a copy of the original.

Import-Module "C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1.Copy\Pester.psd1" invoke-pester -tag Unit -ExcludeTag DevEnvironment -Script .\Test\SimpleRepro.Tests.ps1

5. Possible Solution

6. Context

The goal is to have all our unit tests in our branches run with no dependencies from the test machine - that is all dependencies are part of the branch itself and accessed by path to ensure the same versions of everything are used during testing. This issue means that if anyone has a copy of pester already on their machine, it prevents the copy loaded by path from behaving correctly.

it-praktyk commented 6 years ago

@JamesCrompton-MSFT, I couldnt reproduce the issue that you reported. Can you check commands/outputs provided below and answer me if I understood your report correctly?

Thank you.

PS >$env:PSModulePath
C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules

PS > mkdir C:\Pester-Test
PS > cd C:\Pester-Test

PS > git clone https://github.com/Pester/Pester.git 4.4.0

PS >cd C:\Pester-Test\4.4.0\
PS >git log -n 1
commit b711002ed34843981efaf9ef69d7dcf1cfcaaf60 (HEAD -> master, origin/master, origin/HEAD)
Author: Jakub Jare<C5><A1> <me@jakubjares.com>
Date:   Sun May 6 20:49:22 2018 +0200

    Remove old news about linux compatibility

PS C:\Pester-Test\

PS >ipmo C:\Pester-Test\4.4.0\Pester.psd1
PS >get-module Pester | fl

Name              : Pester
Path              : C:\Pester-Test\4.4.0\Pester.psm1
Description       : Pester provides a framework for running BDD style Tests to execute and validate PowerShell commands inside of PowerShell and offers a powerful set of Mocking Functions that allow tests to mimic and mock the functionalit
                    y of any command inside of a piece of PowerShell code being tested. Pester tests can execute any command or script that is accessible to a pester test file. This can include functions, Cmdlets, Modules and scripts. Pest
                    er can be run in ad hoc style in a console or it can be integrated into the Build scripts of a Continuous Integration system.
ModuleType        : Script
Version           : 4.4.0
NestedModules     : {Gherkin, Axiom, Format, TypeClass}
ExportedFunctions : {Add-AssertionOperator, AfterAll, AfterEach, AfterEachFeature...}
ExportedCmdlets   :
ExportedVariables :
ExportedAliases   : {And, But, Given, Then...}

PS >git clone https://github.com/equelin/format-pester.git
Cloning into 'format-pester'...
remote: Counting objects: 381, done.
remote: Total 381 (delta 0), reused 0 (delta 0), pack-reused 381R
Receiving objects: 100% (381/381), 758.38 KiB | 526.00 KiB/s, done.
Resolving deltas: 100% (178/178), done.

PS >cd .\format-pester\

PS >Invoke-Pester -Script .\demo\DemoFunction1.Tests.ps1 -Tag Random
Executing all tests in '.\demo\DemoFunction1.Tests.ps1' with Tags Random

Executing script .\demo\DemoFunction1.Tests.ps1

  Describing DemoFunction1 - Random

    Context Useless test R-1-1
      [+] does something useful R-1-1-1 185ms
      [+] does something useful R-1-1-2 27ms
      [-] does something useful R-1-1-3 33ms
        Expected the actual value to be less than 1, but got 31.
        25:             DemoFunction1 -FirstParam $(Get-Random -Maximum 100 -Minimum 0) | Should BeLessThan $(Get-Random -Maximum 100 -Minimum 0)
        at Invoke-LegacyAssertion, C:\Pester-Test\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction1.Tests.ps1: line 25
      [?] does something useful R-1-1-4 or Inconclusive 44ms
        Inconclusive result - random - R-1-1-4
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction1.Tests.ps1: line 35
        35:                 Set-TestInconclusive -Message "Inconclusive result - random - R-1-1-4"
      [-] does something useless R-1-1-5 or Inconclusive 35ms
        Expected the actual value to be less than 17, but got 42.
        57:                 DemoFunction1 -FirstParam $(Get-Random -Maximum 100 -Minimum 0) | Should BeLessThan $RandomResult
        at Invoke-LegacyAssertion, C:\Pester-Test\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction1.Tests.ps1: line 57

    Context Useless test R-1-2
      [-] does something  useless R-1-2-1 116ms
        Expected 30, but got 31.
        69:             DemoFunction1 -FirstParam $(Get-Random -Maximum 32 -Minimum 27) | Should Be 30
        at Invoke-LegacyAssertion, C:\Pester-Test\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction1.Tests.ps1: line 69
      [+] does something  useless R-1-2-2 45ms
      [+] does something useful R-1-2-3 22ms
      [-] does something useless R-1-2-4 or Inconclusive 39ms
        Expected the actual value to be less than 1, but got 15.
        98:                 DemoFunction1 -FirstParam $(Get-Random -Maximum 100 -Minimum 0) | Should BeLessThan $RandomResult
        at Invoke-LegacyAssertion, C:\Pester-Test\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction1.Tests.ps1: line 98
      [+] does something useless R-1-2-5 or Pending 60ms
      [-] does something useless R-1-2-6 or Skipped 49ms
        Expected the actual value to be less than 17, but got 68.
        138:                 DemoFunction1 -FirstParam $(Get-Random -Maximum 100 -Minimum 0) | Should BeLessThan $RandomResult
        at Invoke-LegacyAssertion, C:\Pester-Test\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction1.Tests.ps1: line 138
Tests completed in 660ms
Tests Passed: 5, Failed: 5, Skipped: 0, Pending: 0, Inconclusive: 1

PS >Remove-Module Pester

New-Item -Path C:\Pester-Test\Pester -ItemType Directory
Move-Item -Path C:\Pester-Test\4.4.0 C:\Pester-Test\Pester\ -force

PS >$oldPSModulePath = $env:PSModulePath
PS >$env:PSModulePath = "C:\Pester-Test;" + $oldPSModulePath
PS >$env:PSModulePath
C:\Pester-Test;C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules

PS >Get-Module -Name Pester -ListAvailable | Select Name,Version,Path

Name   Version Path
----   ------- ----
Pester 4.4.0   C:\Pester-Test\Pester\4.4.0\Pester.psd1
Pester 4.4.0   C:\Program Files\WindowsPowerShell\Modules\Pester\4.4.0\Pester.psd1
Pester 4.3.1   C:\Program Files\WindowsPowerShell\Modules\Pester\4.3.1\Pester.psd1
Pester 3.4.6   C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.6\Pester.psd1
Pester 3.4.0   C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psd1

Import-Module Pester

PS >Invoke-Pester -Script .\demo\DemoFunction2.Tests.ps1 -ExcludeTag Random
Executing all tests in '.\demo\DemoFunction2.Tests.ps1'

Executing script .\demo\DemoFunction2.Tests.ps1

  Describing DemoFunction2 - Static

    Context Useless test S-2-1
      [+] does something useful S-2-1-1 132ms
      [+] does something useful S-2-2-2 23ms
      [?] doesn't do nothing - is inconclusive S-2-2-3 31ms
        Inconclusive by design 1
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 111
        111:             Set-TestInconclusive -Message "Inconclusive by design 1"
      [?] doesn't do nothing - is inconclusive S-2-2-4 42ms
        Inconclusive by design 2
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 117
        117:             Set-TestInconclusive -Message "Inconclusive by design 2"
      [-] does something useful S-2-2-5 55ms
        Expected 56 to be different from the actual value, but got the same value.
        124:             DemoFunction2 -FirstParam 56 | Should Not Be 56
        at Invoke-LegacyAssertion, C:\Pester-Test\pester\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 124

    Context Useless test S-2-2
      [?] doesn't do nothing - is inconclusive S-2-2-1 158ms
        Inconclusive by design 3
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 135
        135:             Set-TestInconclusive -Message "Inconclusive by design 3"
      [+] does something  useless S-2-2-2 23ms
      [-] does something  useless S-2-2-3 36ms
        Expected the actual value to be greater than 3, but got 2.
        147:             DemoFunction2 -FirstParam 2 | Should BeGreaterThan 3
        at Invoke-LegacyAssertion, C:\Pester-Test\pester\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 147
      [+] does something useful S-2-2-4 59ms
Tests completed in 564ms
Tests Passed: 4, Failed: 2, Skipped: 0, Pending: 0, Inconclusive: 3

PS >Remove-Module Pester

PS >Import-Module C:\Pester-Test\Pester\4.4.0\Pester.psd1

PS >Invoke-Pester -Script .\demo\DemoFunction2.Tests.ps1 -ExcludeTag Random
Executing all tests in '.\demo\DemoFunction2.Tests.ps1'

Executing script .\demo\DemoFunction2.Tests.ps1

  Describing DemoFunction2 - Static

    Context Useless test S-2-1
      [+] does something useful S-2-1-1 235ms
      [+] does something useful S-2-2-2 24ms
      [?] doesn't do nothing - is inconclusive S-2-2-3 21ms
        Inconclusive by design 1
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 111
        111:             Set-TestInconclusive -Message "Inconclusive by design 1"
      [?] doesn't do nothing - is inconclusive S-2-2-4 36ms
        Inconclusive by design 2
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 117
        117:             Set-TestInconclusive -Message "Inconclusive by design 2"
      [-] does something useful S-2-2-5 57ms
        Expected 56 to be different from the actual value, but got the same value.
        124:             DemoFunction2 -FirstParam 56 | Should Not Be 56
        at Invoke-LegacyAssertion, C:\Pester-Test\Pester\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 124

    Context Useless test S-2-2
      [?] doesn't do nothing - is inconclusive S-2-2-1 138ms
        Inconclusive by design 3
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 135
        135:             Set-TestInconclusive -Message "Inconclusive by design 3"
      [+] does something  useless S-2-2-2 28ms
      [-] does something  useless S-2-2-3 42ms
        Expected the actual value to be greater than 3, but got 2.
        147:             DemoFunction2 -FirstParam 2 | Should BeGreaterThan 3
        at Invoke-LegacyAssertion, C:\Pester-Test\Pester\4.4.0\Functions\Assertions\Should.ps1: line 188
        at <ScriptBlock>, C:\Pester-Test\format-pester\demo\DemoFunction2.Tests.ps1: line 147
      [+] does something useful S-2-2-4 44ms
Tests completed in 630ms
Tests Passed: 4, Failed: 2, Skipped: 0, Pending: 0, Inconclusive: 3

PS >Get-Content 'C:\Pester-Test\repro.Test.ps1' -Raw
Describe "pester tests." -Tag DevEnvironment { Context "Sanity test."
{
It "1 should be 1" `
{
1 | should be 1
}
}
}
PS >Invoke-Pester -Script C:\Pester-Test\repro.Test.ps1 -ExcludeTag DevEnvironment
Executing all tests in 'C:\Pester-Test\repro.Test.ps1'

Executing script C:\Pester-Test\repro.Test.ps1
Tests completed in 0ms
Tests Passed: 0, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0

PS >Invoke-Pester .\repro.Test.ps1 -Tag DevEnvironment
Executing all tests in '.\repro.Test.ps1' with Tags DevEnvironment

Executing script .\repro.Test.ps1

  Describing pester tests.
    [-] Error occurred in Describe block 136ms
      RuntimeException: No test script block is provided. (Have you put the open curly brace on the next line?)
      at <ScriptBlock>, C:\Pester-Test\repro.Test.ps1: line 1
      at DescribeImpl, C:\Pester-Test\Pester\4.4.0\Functions\Describe.ps1: line 170
Tests completed in 136ms
Tests Passed: 0, Failed: 1, Skipped: 0, Pending: 0, Inconclusive: 0

PS >Get-Content 'C:\Pester-Test\repro.Test.ps1' -Raw
Describe "pester tests." -Tag DevEnvironment { Context "Sanity test." {
It "1 should be 1" {
1 | should be 1
}
}
}
PS >Invoke-Pester .\repro.Test.ps1 -Tag DevEnvironment
Executing all tests in '.\repro.Test.ps1' with Tags DevEnvironment

Executing script .\repro.Test.ps1

  Describing pester tests.

    Context Sanity test.
      [+] 1 should be 1 113ms
Tests completed in 113ms
Tests Passed: 1, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0
JamesCrompton-MSFT commented 6 years ago

Thank you for your response. I followed your steps and was not able to repro on a new machine, however i still have my original dev machine where this is reproing so the repro is not as simple as i first thought. I have been busy with other things, but i will try and look this weekend at what is special about my repro box and update the issue accordingly.

Thanks, -James

JamesCrompton-MSFT commented 6 years ago

Looks like this was due to strange loading behavior of modules with device guard enabled on my development machine and not due to pester specifically. Closing the issue.

JamesCrompton-MSFT commented 6 years ago

Looks like this was due to strange loading behavior of modules with device guard enabled on my development machine and not due to pester specifically. Closing the issue.