Jaykul
commented
5 years ago Side note: Can you edit the original post to put
codemarkers around things? Specifically, where you had wildcards in the docs for path and filters -- it seems that a few backslashes and wildcards are missing in the rendered output.
I like the idea of simplifying the core Pester command, even if it means we need a "Filter" or "Options" command.
I wonder if a config file, rather than (just) a live object would be ideal for helping contributors run tests the way I expect them to be run against my projects. Maybe the Filter command should be called New-PesterConfig and can return an object you can pass to a -Config parameter -- or can write a PesterOptions.psd1 file and you can pass the path to that, instead. Pester could even look for PesterOptions.psd1 in the test path you pass to it, picking it up automatically.
In any case, the hard part is getting the right "minimal" set of options.
Excluding things
I think that "Exclude" for anything but Tag should be an advanced feature. Partly just because interactive use would be so much better if there was only one parameter starting with -Ex, and partly because I can sum up the vast majority of my own use as "run all the tests in this folder except the ones that I tagged so that I could exclude them" E.g.:
Invoke-Pester -ExcludeTag SqlIntegrationFor all of my other filtering, I can get by with an included folder list and/or specifying the test names.
Continuous Integration
I can't imagine running CI builds without measuring code coverage, and exporting the test results and code coverage to files so that that we can upload them to the CI system as results. Perhaps rather than a simplistic -CI switch, we need a wrapper around Pester for integration runs, so we can avoid needing to -Passthru the object and pass that to multiple export commands.
In addition to handling the source path for coverage, and the output paths for both coverate and results, a CI wrapper need to handle updating the PSModulePath for the test environment
My teams and projects are moving to using tools for dependencies. RequiredModules and PSDepend support saving required modules to a sub-folder. But during testing I need to fix that folder at the front of my PSModulePath, along with the location of my code under test. Particularly important for Pester to handle if it's going to try to parallelize or watch with runspaces/jobs.
Watching
I'm not too sure about this scenario.
In an ideal -Watch scenario (i.e. the way "continuous testing" works in Visual Studio), I want the tool to understand source control (what code has changed) and test coverage (which tests cover which code) and prioritize tests that affect stuff that's changed. Then it needs to feed the results back to the editor so it can provide continuous feedback to the user, like NCrunch or VS2017
For now, I this this should be a separate command, because it's incomplete without the test coverage correlation to edits, and will rarely be used until there is a tighter feedback loop (i.e. editor integration).
Anecdotally: we wrote a watching wrapper at work a few years ago, which guessed at which tests to run based on pending edits and test file names or tags. Despite the effort put in (and the knowledge of our code structure which made it possible to filter which tests to run) we rarely used it because the results were not visible in the editor where we were working. By the time we switched to git, nobody even bothered updating it.
One note from that experience: for PowerShell, you need to at least re-import the code under test -- otherwise the user has to unload/reload in their test fixtures.
nohwnd
felixfbecker
fflaten
I am trying to simplify the interfaces on Invoke-Pester, to create a pit of success, where the usual stuff is simple easy to discover, and advanced stuff is less simple and less easy to discover. The way Pester v4 does that (to some extent) is by hiding hashtables behind parameters, this gives you a lot of freedom, but hardly a clean strongly typed interface. What I would like is a simple interface to be easy to start with, and the configurations to be explicit so I don't have to remember structure of hashtables. It would still be nice to have hashtables, so they can be used as literals in the code, modified and passed around easily.
Simple
What I am assuming is that a typical person has a folder with tests, and want's to do some basic filtering on them. They also want to run them locally to develop, and run them on a build server easily. To me the simplest non restrictive parameter set is:
This would cover 99% of what I do. 😁
This minimal api would then be typically used like this:
Invoke-PesterList of paths to run*.Tests.ps1from. When the path is a folder it will run all*.Tests.ps1files in it and its subfolders. When the path is a wildcarded file likec:\tests\*.ps1it will run all*.Tests.ps1in that folder, if that path is a full path, it will run the path, even if it is not a.Tests.ps1file. Defaults to the current directory "." (Just like in v4).-Path <string[]>Excludes paths from the test run, it uses -like wildcards to match the paths, and automatically replaces '\' to '/' to make it work cross-plaform easily.
-ExcludePath <string[]>Filters on Block and Test name, uses
.as a delimiter for the path. I think this is a useful and easy to use convention. For example having a test 'it1' in 'describe2'that is inside of 'describe1' would give the test a full path ofdescribe1.describe2.it1, and we can then filter like this: 'it1' to run just it, ordescribe1.*to run everything in that block, or*.describe2.*to run everything in describe2 block no matter where it is placed.-Name <string[]>Same as above, but it excludes the tests that match. Exclude takes precedence before any other filter.-ExcludeName <string[]>Filters blocks and tests based on their tags. If any tag matches the test will run. Accepts like wildcards.
-Tag <string[]>Same as above but it excludes tags.-ExcludeTag <string[]>Outputs the test result
-PassThru <switch>Runs the tests in parallel (with per-file granularity), the number allows you to specify how many tests should run in parallel, defaults to your core count.
-Parallel <int>Watch the given file paths, and if any file changes wait 2s and then run the tests in it
-Watch <switch>A single switch that combines multiple things that are used in CI builds. Run the tests in strict mode (only passed/failed results are permitted but not Skip etc.). Do NOT focus tests/blocks. Exit with exit code if any test fails.
-Ci <switch>This controls the verbosity of the on-screen output. The naming clashes with what is already there, and the goal is not to give all the options as in v4 here, because there might me more options, the output should be customizable, and because different modes (e.g. normal run, vs a parallized watch run) might need different output formatting to make sense. So say we go with the first:
-Output Normal, Short, None (or -Show?)Then Normal would show, the complete output. Short would show the failed tests that passed per block and then it would print failed tests.
There is still option to add more verbosity before (like Trace, Debugging, Detailed) and in the middle, like Minimal, but let's not do that now, especially with the possibility that the output will be customizable.
Other possible namings
-Output Detailed, Normal, Minimal, Quiet-Output Trace, Debugging, Normal, Short, Minimal, NoneAdvanced
For advanced stuff there would be functions that would generate the objects you need to provide, that way we avoid having confusing api that uses Object for parameters just so you can provide a string / string[] / scriptblock / hashtable / hashtable with aliases to a
-Scriptparameter that is used as-Path <string>90% of the time anyway.So for example defining an advanced filter would be:
The New-Filter would produce an object that would contain all the filter parameters you specified. This has few advantages over pure hashtables:
There might also be switches that would return the object as a hashtable / template so you can manipulate it easily by code, or write it as a literal and then splat it.
e.g.
New-Filter -AsHashtableLiteral | clipThe same approach would be applied to other advanced stuff like:
It would also be a nice step towards configuration because
In between?
There is probably a huge gap between the simple interface, and the advanced interface, is there another one needed? How would it look like? How it would be implemented to not overwhelm the beginner trying to understand what to do?