search

pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

Some tests fail when Set-StrictMode is set to Latest #870

Closed it-praktyk closed 7 years ago

it-praktyk commented 7 years ago

Some tests fail when Set-StrictMode fail when Set-StrictMode is set to Latest.

Pester version     : 4.0.6 <FOLDER_PATH>\Pester\Pester.psm1 - the branch Dev4-1_PSCore_compatibility
PowerShell version : 5.1.15063.502
OS version         : Microsoft Windows NT 10.0.15063.0

Most of the errors are related to Gherkin-style tests under PowerShell 2.0 - especially under execution by Travis-based continues integration.

To reproduce you can add Set-StrictMode at the beginning of Pester.psm1.

Executing script <FOLDER_PATH>\Pester\Functions\Coverage.Tests.ps1

  Describing Code Coverage Analysis

    Context Entire file
      [-] JaCoCo report must be correct 54ms
        PropertyNotFoundException: The property 'Count' cannot be found on this object. Verify that the property exists.
        at Get-JaCoCoReportXml, <FOLDER_PATH>\Pester\Functions\Coverage.ps1: line 601
        at <ScriptBlock>, <FOLDER_PATH>\Pester\Functions\Coverage.Tests.ps1: line 76

    <OUTPUT_PARTIALLY_OMMITTED>

  Describing Stripping common parent paths
    [-] Error occurred in Describe block 63ms
      RuntimeException: The variable '$IsLinux' cannot be retrieved because it has not been set.
      at <ScriptBlock>, <FOLDER_PATH>\Pester\Functions\Coverage.Tests.ps1: line 237
      at DescribeImpl, <FOLDER_PATH>\Pester\Functions\Describe.ps1: line 161

The variable '$StepFiles' cannot be retrieved because it has not been set.

``` - [ ] Pester\Functions\Gherkin.Tests.ps1 - case 2 ```Pester\Functions\Gherkin.Tests.ps1 Executing script \Pester\Functions\Gherkin.Tests.ps1 The property 'Name' cannot be found on this object. Verify that the property exists. + CategoryInfo : NotSpecified: (:) [], PropertyNotFoundException + FullyQualifiedErrorId : PropertyNotFoundStrict + PSComputerName : localhost The property 'Name' cannot be found on this object. Verify that the property exists. + CategoryInfo : NotSpecified: (:) [], PropertyNotFoundException + FullyQualifiedErrorId : PropertyNotFoPester\Functions\Gherkin.Tests.ps1 + PSComputerName : localhost ``` - [ ] Pester\Functions\Gherkin.Tests.ps1 - case 3 ```Pester\Functions\Gherkin.Tests.ps1 Executing script \Pester\Functions\Gherkin.Tests.ps1 The property 'Tags' cannot be found on this object. Verify that the property exists. + CategoryInfo : NotSpecified: (:) [], PropertyNotFoundException + FullyQualifiedErrorId : PropertyNotFoundStrict + PSComputerName : localhost ``` - [x] Pester\Functions\Mock.Tests.ps1Pester\Functions\Mock.Tests.ps1 ``` Executing script \Pester\Functions\Mock.Tests.ps1 Describing When calling Mock on an external script [-] Error occurred in Describe block 85ms PropertyNotFoundException: The property 'Body' cannot be found on this object. Verify that the property exists. at Get-DynamicParamBlock, \Pester\Functions\Mock.ps1: line 1297 at Mock, \Pester\Functions\Mock.ps1: line 245 at , \Pester\Functions\Mock.Tests.ps1: line 172 at DescribeImpl, \Pester\Functions\Describe.ps1: line 161 Describing Single quote in command/module name [-] Command with single quote in name should be mockable 37ms PropertyNotFoundException: The property 'Body' cannot be found on this object. Verify that the property exists.Pester\Functions\Output.Tests.ps1Pester\Functions\Output.Tests.ps1 at Get-DynamicParamBlock, \Pester\Functions\Mock.ps1: line 1297 at Mock, \Pester\Functions\Mock.ps1: line 245 at , \Pester\Functions\Mock.Tests.ps1: line 1715 ``` This test fails because I've used `Invoke-Pester -Show Describe,Context,Failed` - [x] Pester\Functions\Output.Tests.ps1 ``` Executing script \Pester\Functions\Output.Tests.ps1 Describing Write-PesterStart [-] uses Format-PesterPath with the provided path 79ms Expected Format-PesterPath in module Pester to be called at least 1 times but was called 0 times 186: Assert-MockCalled Format-PesterPath -ParameterFilter {$Path -eq $expected} at , \Pester\Functions\Output.Tests.ps1: line 186 ```
it-praktyk commented 7 years ago

@nohwnd, @dlwyatt

Should all tests for Pester itself (Functions*.Tests.ps1 files) start with a declaration Set-StrictMode -Version Latest?

I've found that the Functions\Coverage.Tests.ps1 doesn't contain this.

nohwnd commented 7 years ago

yup. whole pester should be able to run in strict mode.

it-praktyk commented 7 years ago

OK, I'll update style rules tests to check if the first line of Functions*.Tests.ps1 is Set-StrictMode -Version Latest to avoid this kind of situation in the future.

BTW, I can't find any tests for the Set-TestInconclusive function.

nohwnd commented 7 years ago

Okay maybe not this way. Once the strict mode is set you don't have to re-set it for every file right? So this is likely just a remnant of past. No need to add more rules for this, imho the code on build server runs in strict mode anyway, so it would get caught there. If it is not we should set it up that way.

it-praktyk commented 7 years ago

Hmmm, I caught errors provided above after when I set Set-StrictMode -Version Latest just at the beginning of my local copy of the Pester.psm1. They were not caught by CI.

nohwnd commented 7 years ago

In that case I am wrong about how Strict mode works or something else is wring. I'll have a look when I have some spare time.

it-praktyk commented 7 years ago

The list of tests files which don't start from Set-StrictMode

PS Pester> Get-ChildItem -Include *.Tests.ps1 -Recurse | foreach { [String]$FileFirstLine = "{0} - {1}" -f $_.Name, $(Get-Content $_ | Select-Object -First 1) ; If ( $FileFirstLine -notmatch 'StrictMode') { Write-Host $FileFirstLine } }

BreakAndContinue.Tests.ps1 - Describe 'Clean handling of break and continue' {
Environment.Tests.ps1 - InModuleScope -ModuleName Pester {
GlobalMock-A.Tests.ps1 - # This script exists to create and mock a global function, then exit.  The actual behavior
GlobalMock-B.Tests.ps1 - # This test depends on some state set up in GlobalMock-A.Tests.ps1.  The behavior we're verifying
New-MockObject.Tests.ps1 - describe 'New-MockObject' {
Output.Tests.ps1 - InModuleScope -ModuleName Pester -ScriptBlock {
SetupTeardown.Tests.ps1 - Describe 'Describe-Scoped Test Case setup' {
TestsRunningInCleanRunspace.Tests.ps1 - function Invoke-PesterInJob ($ScriptBlock, [switch] $GenerateNUnitReport)
Pester.Tests.ps1 - $here = Split-Path -Parent $MyInvocation.MyCommand.Path
it-praktyk commented 7 years ago

Guys, I need help!

I stacked in troubleshooting of the current issue. The whole what I did is in the branch https://github.com/it-praktyk/pester/tree/Dev4-1_PSCore_compatibility.

Thank you in advance.

CC: @Jaykul

it-praktyk commented 7 years ago

@nohwnd, I don't understand what the line does.

When I've added Set-StrictMode -Latest at the beginning of the Pester.psm1 $PSVersionTable.ClrVersion$PSVersionTable.ClrVersiont$PSVersionTable.ClrVersion$PSVersionTable.ClrVersion$PSVersionTable.ClrVersionhe error like below is thrown.

Could you look on this, please?

    [-] Should validate test results against the nunit 2.5 schema 160ms
      Expected: the expression not to throw an exception. Message was {Exception calling "Validate" with "1" argument(s): "The property 'Exception' cannot be found on this object. Verify that the property exists."}
          from <FOLDER_PATH>\Pester\Functions\TestResults.Tes$PSVersionTable.ClrVersion$PSVersionTable.ClrVersion$PSVersionTable.ClrVersionts.ps1:148 char:19
          +                 { $xml.Validate({throw $args.Exception }) } | Should  ...
          +                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      148:                 { $xml.Validate({throw $args.Exception }) } | Should Not Throw
      at Invoke-LegacyAssertion, <FOLDER_PATH>\Pester\Functions\Assertions\Should.ps1: line 190
      at <ScriptBlock>, <FOLDER_PATH>\Pester\Functions\TestResults.Tests.ps1: line 148

[Update] Previously I disabled the line because $PSVersionTable.ClrVersion is not available on PSCore.

When I've rerun tests without set Set-StrictMode the error message was clear - the xsd schema violation.

it-praktyk commented 7 years ago

3 days ago I've asked for help to resolve the issue #870. I was not clear what does it mean and what I really need.

I tried to resolve issues with not initialized variables in the Gherkin related code - some examples are above. Due to lack of time, I can't learn Gherkin notations and analyze this part of the code.

All your help will be appreciated. Thank you.

it-praktyk commented 7 years ago

Most of the issues are resolved and proposed to merge as a part of the pull request #925.

Gherkin related are sent back to the @Jaykul's fork as a part of the pull request https://github.com/Jaykul/Pester/pull/1 - the code what caused them was not merged to Pester yet.