Pester fails during code coverage setup if a hidden directory is included in CodeCoverage.Path

martincostello commented 1 year ago

Checklist

[X] Issue has a meaningful title
[X] I have searched the existing issues. See all issues
[X] I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

I have a project where I'm using pester where I use a dot-prefixed folder to provide a sample in a directory to ensure it sorts first above a sequential list of numbers for other directory names, for example:

.sample
1
2
3
<etc>

If I explicitly add all folders into the CodeCoverage.Path setting of the Pester configuration like so, specifying -Force to ensure hidden folders are returned (because of the . prefix):

$config.CodeCoverage.Path = Get-ChildItem -Path (Join-Path $PSScriptRoot "migrations") -Directory -Force | ForEach-Object { $_.FullName }

Then Pester fails before tests start to run with an error similar to the below:

Pester v5.4.0

Starting discovery in 2 files.
Discovery found 14 tests in 234ms.
Starting code coverage.
Invoking step RunStart failed:
Result :Error :
Get-Item: Could not find item
/runner/_work/project-name/project-name/migrations/.sample.

at Get-CodeCoverageFilePaths, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 85[9](https://github.je-labs.com/codeflow/repo-migrations/runs/1440788?check_suite_focus=true#step:5:10)1
at Resolve-CoverageInfo, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 8566
at Get-CoverageInfoFromUserInput, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 8497
at Enter-CoverageAnalysis, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 8376
at <ScriptBlock>, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 8317
at Invoke-PluginStep, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 1773
at Run-Test, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line [15](https://github.je-labs.com/codeflow/repo-migrations/runs/1440788?check_suite_focus=true#step:5:16)73
at Invoke-Test, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 2475
at Invoke-Pester<End>, /home/runner/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1: line 5272
at <ScriptBlock>, /runner/_work/repo-migrations/repo-migrations/test.ps1: line [19](https://github.je-labs.com/codeflow/repo-migrations/runs/1440788?check_suite_focus=true#step:5:20)

Expected Behavior

Pester successfully runs code coverage and the tests.

Steps To Reproduce

No response

Describe your environment

From WSL on Windows 11 running Ubuntu 22.04:

Pester version     : 5.4.0 /home/martin/.local/share/powershell/Modules/Pester/5.4.0/Pester.psm1
PowerShell version : 7.3.3
OS version         : Unix 5.15.90.1

This also occurs in a Linux container running Ubuntu in GitHub Actions, which is where I originally found the issue as I have a Windows development machine and the issue was not apparent there due to the different handling of folders with a . prefix.

Possible Solution?

From a cursory look at the code pointed to by the stack trace, it looks like -Force would needed to be passed through to this call to Get-Item so the directories/files are found?

https://github.com/pester/Pester/blob/d81fdbc46318d6c3486c241fa82d9749df5c2031/src/functions/Coverage.ps1#L231

fflaten commented 1 year ago

Good catch! Looks like we currently ignore hidden items for both test files and CC paths, including hidden files when a folder is specified. What would be the desired behavior?

1) Only include hidden items that are explicitly named? 2) Always include hidden items, including hidden files when a folder is specified

Leaning towards option 1 myself as I'd expect there's a reason someone would make items hidden in the first place. 🙂

martincostello commented 1 year ago

1 sounds sensible to me.

tbutler-qontigo commented 1 month ago

Hi I think that Pester should behave consistently across operating systems - I reported in #2515 that we see tests/.github tests being recognised on windows but not on linux.

A disadvantage or your item 1 suggest is that this means that all test folders following the recommended structure will have to be explicitly named, which makes it more of a chore. For example if I have this structure

.github
foo
tests
   \.github
   \foo

Then currently, on windows I can either not specify a location at all and all my tests will run or simply Invoke-Pester tests to get all the tests under tests to be found. If the only way I can get my .github tests to be discovered on linux is to specify them at the command line then either I have to Invoke-Pester tests,tests/.github in which case they run twice, which is not desirable, or I have to Invoke-Pester tests/foo,tests/.github which means if I now want to add folder bar then I have to remember to add this one to the command line explicitly, which is error prone as it relies on prior knowledge of the engineer to "just know" to have to do this.

Workable options for me would include:

Always find files in tests/.github whether a folder is specified or not, possibly with an option to disable test discovery in "hidden" folders
Only find files in tests/.github whether a folder is specified or not if a command line option specifically enables searching in a hidden folder.
If you specify a folder at all then everything below it would be found - so Invoke-Pester would not find tests/.github but Invoke-Pester tests would

My preference is for the first option, but regardless of which one is selected, the behaviour should be consistent across operating systems. Our use case is we are writing scripts which might be run on windows or on linux so we want to test them in both operating systems.

Thanks

pester / Pester