Pester 5 tests seems to run twice when paths are not specified in build.yml

pjstam commented 2 years ago

Problem description

Hi there, It looks like the pester tests run twice when the Path: is left default (commented). Workaround is to specify the paths. Regards, Peter

Pester:
  OutputFormat: NUnitXML
  # Excludes one or more paths from being used to calculate code coverage.
  ExcludeFromCodeCoverage:

  # If no scripts are defined the default is to use all the tests under the project's
  # tests folder or source folder (if present). Test script paths can be defined to
  # only run tests in certain folders, or run specific test files, or can be use to
  # specify the order tests are run.
  Path:
  # - tests/QA/module.tests.ps1
  - tests/QA **when commenting, it runs twice**
  - tests/Unit **when commenting, it runs twice**
  # - tests/Integration
  ExcludeTag:
  #  - helpQuality
  #  - FunctionalQuality
  #  - TestQuality

Verbose logs

The output shows the run twice.

How to reproduce

Create a New-SampleModule with 'simplemodule' type.

Expected behavior

It should run once.

Current behavior

It's running twice.

Operating system the target node is running

OsName               : Microsoft Windows 10 Enterprise
OsOperatingSystemSKU : EnterpriseEdition
OsArchitecture       : 64-bit
WindowsVersion       : 2009
WindowsBuildLabEx    : 19041.1.amd64fre.vb_release.191206-1406
OsLanguage           : en-US
OsMuiLanguages       : {en-US, en-GB}

PowerShell version and build the target node is running

Name                           Value
----                           -----
PSVersion                      7.2.0
PSEdition                      Core
GitCommitId                    7.2.0
OS                             Microsoft Windows 10.0.19043
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module version used

Latest  0.112.0

gaelcolas commented 2 years ago

I think by default it should run the tests in tests/ folder. Could you share the output? Especially interested in the task headers that should tell what are the path resolving to.

gaelcolas commented 2 years ago

Also, if your project is a DSC Community project (running HQRM tasks) then some tests that are under QA might also be duplicated in those HQRM tests. Having the logs would help know what happened.

pjstam commented 2 years ago

Hi @gaelcolas , It looks like the line at 875/876 in the file Invoke-Pester.pester.build.ps1 is causing this issue (task _v5). There is a double Join-Path, one with capital T -> 'Test' and one with t -> 'test' (probably done because of a case sensitive filesystem). A bit further at line 884 there is a loop that loops through the configured paths in the build.yml. So when uncommenting just one path the loop takes place one time, thus one test sequence. When leaving the build.yml default (everything commented) the double Join-Path is applied, thus running the tests twice.

Regards, Peter

johlju commented 2 years ago

Running the tests on Linux and passing the wrong case than the actual folder name or filename Pester seems to handle it. If I pass Tests when the actual folder is test, Pester finds the tests.

So I think it is safe just to remove one the later Join-Path (line 876).

gaelcolas / Sampler