pester / Pester

Pester is the ubiquitous test and mock framework for PowerShell.
https://pester.dev/
Other
3.11k stars 473 forks source link

State of the help/documentation #1634

Closed jhoneill closed 5 months ago

jhoneill commented 4 years ago

1. General summary of the issue

Pester's documentation is not up to the same standard as the rest of the module.

2. Describe Your Environment

PowerShell 5, 7.0.3, Windows Terminal, VS Code,

3. Expected Behavior

  1. Should -? gives valid / useful help.

  2. get-help about_Should works

  3. Invoke-pester -? returns parameter help.

etc.

4.Current Behavior

should -?
...
SYNTAX
    Should [[-ActualValue] <Object>] [<CommonParameters>]
...
RELATED LINKS
    https://github.com/pester/Pester/wiki/Should
    about_Should
    about_Pester

REMARKS
    To see the examples, type: "get-help Should -examples".

The remarks are added by the help system but there are no examples. The URL is no longer current. Help for a command shouldn't be in an about topic

>get-help about_should

Name                              Category  Module                    Synopsis
----                              --------  ------                    --------
about_Should                      HelpFile                            Provides assertion convenience methods for comparing objects and throwing
about_should                      HelpFile                            Provides assertion convenience methods for comparing objects and throwing

This may be a consequence of how about_ help is processed and having multiple pester versions present. Because of syntax changes there may be old and new versions present on a system, VS code installs a version in parallel with any the user has etc.

Invoke-Pester seems to be missing help entirely.

5. Possible Solution

It needs input from more than one person (and I'm happy to be one of the people) to fill in the gaps, make corrections, and generally bring the help up the same standard as a the code. As with PowerShell itself this should include people who wouldn't consider adding to the code.

asears commented 4 years ago

Using Pester 5.02 / Powershell 5.1.18362.752 and happy to help with this too, though it's a pretty broad issue to create examples for each command and duplicate the effort both online and in the offline help. I wonder what other module projects have done to automate / synch this?

I see this coming up in PSScriptAnalyzer recurse scan.

ProvideCommentHelp                Information  Pester.Run 78    The cmdlet 'Find-Test' does not have a help comment.
PSProvideCommentHelp                Information  Pester.Run 746   The cmdlet 'Is-Discovery' does not have a help comment.
PSProvideCommentHelp                Information  Pester.Run 1000  The cmdlet 'Invoke-PluginStep' does not have a help comment.
PSProvideCommentHelp                Information  Pester.Run 1680  The cmdlet 'Invoke-Test' does not have a help comment.
PSProvideCommentHelp                Information  Pester.Run 2372  The cmdlet 'Recurse-Up' does not have a help comment.
PSProvideCommentHelp                Information  Pester.Moc 345   The cmdlet 'Public' does not have a help comment.
PSProvideCommentHelp                Information  InModuleSc 31    The cmdlet 'PublicFunction' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 458   The cmdlet 'PublicFunction' does not have a help comment.
                                                 .ps1             not have a help comment.
                                                 .ps1             have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 481   The cmdlet 'PublicFunction' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 484   The cmdlet 'PublicFunction2' does not have a help comment.
                                                 .ps1             have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 494   The cmdlet 'ScopeTest' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 1046  The cmdlet 'PublicFunction' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 1340  The cmdlet 'PublicFunction' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 1352  The cmdlet 'PublicFunction' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2203  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2252  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2265  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2441  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2504  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2555  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2565  The cmdlet 'f' does not have a help comment.
PSProvideCommentHelp                Information  Mock.Tests 2581  The cmdlet 'f' does not have a help comment.

Help works for Invoke-Pester on this machine

get-help invoke-pester

NAME
    Invoke-Pester

SYNTAX
    Invoke-Pester [[-Path] <string[]>] [-ExcludePath <string[]>] [-TagFilter <string[]>] [-ExcludeTagFilter <string[]>] [-FullNameFilter <string[]>] [-CI] [-Output {Diagnostic | Detailed
    | Normal | Minimal | None}] [-PassThru]  [<CommonParameters>]

    Invoke-Pester [[-Path] <string[]>] [[-FullNameFilter] <string[]>] [[-EnableExit]] [[-TagFilter] <string[]>] [-ExcludeTagFilter <string[]>] [-PassThru] [-CodeCoverage <Object[]>]
    [-CodeCoverageOutputFile <string>] [-CodeCoverageOutputFileEncoding <string>] [-CodeCoverageOutputFileFormat {JaCoCo}] [-Strict] [-OutputFile <string>] [-OutputFormat {NUnitXml |
    JUnitXml}] [-Quiet] [-PesterOption <Object>] [-Show {None | Default | Passed | Failed | Pending | Skipped | Inconclusive | Describe | Context | Summary | Header | Fails | All}]
    [<CommonParameters>]

    Invoke-Pester [-Configuration <PesterConfiguration>]  [<CommonParameters>]

ALIASES
    None

REMARKS
    None

Should -?

NAME
    Should

SYNOPSIS
    Should is a keyword what is used to define an assertion inside It block.

SYNTAX
    Should [[-ActualValue] <Object>] [<CommonParameters>]

DESCRIPTION
    Should is a keyword what is used to define an assertion inside the It block.
    Should provides assertion methods for verify assertion e.g. comparing objects.
    If assertion is not met the test fails and an exception is throwed up.

    Should can be used more than once in the It block if more than one assertion
    need to be verified. Each Should keywords need to be located in a new line.
    Test will be passed only when all assertion will be met (logical conjuction).

RELATED LINKS
    https://github.com/pester/Pester/wiki/Should
    about_Should
    about_Pester

REMARKS
    To see the examples, type: "get-help Should -examples".
    For more information, type: "get-help Should -detailed".
    For technical information, type: "get-help Should -full".
    For online help, type: "get-help Should -online"

Examples should show examples, but currently doesn't. The online help command redirects to another page and could have the link fixed to direct to correct link.

Some more details on the Pester 5 configuration object, discovery, code coverage and example scripts and documentation for Devops pipelines would be super useful.

jhoneill commented 4 years ago

Using Pester 5.02 / Powershell 5.1.18362.752 and happy to help with this too, though it's a pretty broad issue to create examples for each command and duplicate the effort both online and in the offline help. I wonder what other module projects have done to automate / synch this?

It's not trivial, for sure



Help works for Invoke-Pester on this machine
That's the base "system help" but there is nothing written to explanation the parameters, give examples, etc just what Get-command returns, formatted in help style. 
nohwnd commented 4 years ago

The online docs are mostly generated based on the comment based help, we just need to set it up to generate also from v5.0 branch.

I plan to start on the docs soon, are there any areas that you'd be interested in helping with?

asears commented 4 years ago

With some guidance would like to help with Mocking documentation and some of the examples in the code. I think the most powerful feature of Pester is its mocking functionality and its understanding of underlying Powershell modules. This helps a lot when things like Azure Powershell change API signatures.

I think it would be hard to update the documentation without a code scanner for the examples, to identify legacy syntax. Could a list of keywords be provided for legacy syntax, their Pester 5 mappings, and list of APIs to document/currently documented as a text file in the repo? This could then be used to search the codebase and generate a report, similar to PsScriptAnalyzer warnings.

How do we automatically "test" (perhaps using Pester?) that the documentation is up to standard? What is the standard for Powershell Get-Help?

A few ideas for tests.

  1. Test that the remarks or additional help keywords which are added by the help system have examples.
  2. Test that the URL in a URL link is a Pester-supported URL, not 404 and text in the page uses Pester 5 syntax.
  3. Test that there are no external URLs from 3rd-party sites in the offline help.
  4. Test that help for a command shouldn't be in an about topic

Perhaps pulling the code within documentation out as separate PS1 files then including afterwards would make it easier to debug/test on each build. And external URLs outside of the Pester wiki could be moved to the repo or documentation web site so they can be scanned for 404's / security and curated for relevance.

eg. get-help about_Mocking shows legacy code & legacy examples & blog posts. These are still helpful but probably shouldn't be in the main offline help and should be scanned regularly for security & 404s.

An ideas for managing this as part of the Github workflow

  1. Label issues / PRs indicating doc update required
asears commented 4 years ago

There are a few "quick hits" here for Migration Guides - Pester v4 to v5 - can reference some materials from issues on the project? Adding a label to tag these issues and surface as changes to document within Pester 5 would make sense?

Test Results, Report Generator & Code Coverage are probably priority items to document further to ease experience implementing in Azure Devops and other CI tools.

ThomasNieto commented 4 years ago

I can also help with the documentation. This would be a good time to use PlatyPS to be able to ship updatable help.

asears commented 4 years ago

I made some progress on the PR, updating synopsis and pulling in some configuration properties, tagging some parameters as deprecated.

Will need some assistance implementing PlatyPS in the build. Probably separate PR.

nohwnd commented 4 years ago

I don't think we need PlatyPS, there was a PR for it open (or maybe still is) for few years, but it did not go anywhere. We already generate all the command related content for pester.dev from the comment based help. What more would we get from platyps?

nohwnd commented 4 years ago

https://github.com/pester/docs/pull/54#issuecomment-683792791 will add v5 docs (and a dropdown to switch to v4) and then I will move the stuff from readme there, and revert back to the normal readme.

Then we can finally start grinding on the new docs. It's been a long time overdue.

asears commented 4 years ago

See #720 for platyps comments. I guess you removed all the markdown /docs previously. I think they're useful for browsing the documentation from Github and would fit well in currently almost empty /docs folder. Or restructure the docs folder itself since it doesn't really contain docs.

If there is already a pattern / tool in place, is there a document identifying the steps to update the comment docs as part of the PR merge / release?

Including some docs outside of comment-based help would also be beneficial - for common setup in Azure Devops / GitHub pipelines for example... There's a lot of info which could be surfaced from the PRs and Issues.

jhoneill commented 4 years ago

I don't think we need PlatyPS, there was a PR for it open (or maybe still is) for few years, but it did not go anywhere. We already generate all the command related content for pester.dev from the comment based help. What more would we get from platyps?

Two or three things.

  1. You can have help in multiple languages.
  2. You get better spelling and grammar checking tools outside of code editors.
  3. It makes the files with the code it a lot smaller (when we did it for importExcel there were more bytes of help text than code).

None of those is overwhelming but big projects tend to outgrow comment based help (which makes much more sense when starting).

bravo-kernel commented 4 years ago

Folks, we are currently using https://docusaurus-powershell.netlify.app/docs/faq/multi-line-examples. This basically is PlatyPS on steroids for multi-line code-fence support.

bravo-kernel commented 4 years ago

It auto-generates the online command reference with Get-Help pages for all Pester functions.

E.g. https://pester.dev/docs/commands/Invoke-Pester

asears commented 4 years ago

Seems much better than PlatyPS. Is it possible to generate markdown to /docs folder as part of the release process to get the markdown help back? Assuming it would be generated in synch with the web docs.

Can the docusaurus step be documented in the repo with steps to generate the docs? I didn't see anything in the devops pipeline. Have started this in #1647 Build.md

Is there a template or set of standards for the about_help in Pester or should put one together?

bravo-kernel commented 4 years ago

@asears in all honesty it's still only a PlatyPS post-processor, even though it is pretty good.

Let me try to give an overview:

  1. there is a PR open at pester/docs that implements the v4 and v5 version (PR link, site preview link). This will put in place the required docusaurus structure with "frozen" documentation for version4 and "current" documentation for v5. See the version dropdown in the preview, top-left

  2. That v5 preview has a "command reference" specific to Pester v5, produced by parsing the Pester v5 Get-Help.

  3. There is already a file describing the steps to generate above command reference, see this file. You can use that file to give it a go locally.

  4. We have always wanted to integrate it with the Pester build pipeline. However, nobody ever picked up that task, we need help there. See this ticket.

What next?

  1. @nohwnd needs to first merge the v4/v5 PR
  2. we can then both:
    • move static readme content to proper v5 website pages (with Edit this Page button)
    • fix any get-help-page rendering issues that might exist
    • get that pester CI pipeline in place to create an "Updated Command Reference" pr in the pester/docs repo
nohwnd commented 4 years ago

Thanks guys, yet another time I am the slow link here, wanted to work on docs this weekend but got caught up into adding profiler to powershell

Will try to review and get the v4/v5 PR merged by tomorrow.

bravo-kernel commented 4 years ago

Folks, the pester.dev website has been updated and now services the v5 documentation.

Requires some love and help to e.g.:

asears commented 4 years ago

I have a draft PR #1647 for the second issue and can help with this. Should add another issue to move current readme information (and other github issue info) into the v5 documentation? There are some rather large github issue threads containing lots of detail which could be of use in the docs.

Some items I noticed so far.

  1. Google Sitemap / Bing Search probably needs a refresh. Getting some invalid links in search results with inurl:https://pester.dev new-pesteroption as example.
  2. Couldn't find these search terms which are part of code and point to wiki.
new-pesteroption
BeforeDiscovery
In
It
BeforeEach-and-AfterEach
BeforeEach
AfterEach
BeforeAll
AfterAll
  1. SYNOPSIS Missing .LINK property (some may be internal)

    Add-ShouldOperator
    BeforeDiscovery
    In
    InModuleScope
    Set-DynamicParameterVariable
    New-MockObject
    Should-InvokeVerifiable
    Should-Invoke
    Should-InvokeVerifiable
    Set-ItResult
    Should-Be (and most of the assertions)
  2. Missing SYNOPSIS

    Get-AllMockBehaviors
    Get-VerifiableBehaviors
  3. As part of PR for new features or contributor guidelines, should a checklist be put together, should it include documentation updates?

  4. Should wiki links be removed from all code after replacement with pester.dev links?

  5. Should some custom tests be put in for missing .LINK, missing .SYNOPSIS etc as part of the code PR? I see Pester.Tests.ps1 already contains some of this. Is there a list reproducing above lists in the build?

  6. I see commit comments regarding .LINK needing to be 1 per link. Should multiple links be separated into 1 per .LINK? eg. image should be

.LINK
about_should
.LINK
about_pester
bravo-kernel commented 4 years ago

@asears thanks for taking the time to look into this with so much detail. I cannot answer all questions but hope this is a good enough start to get you going. Any contribution is more than welcomed. ❤️

0. General

Turning content in "large" github issue conversations into a web page is a very good idea. Feel free to submit a PR.

1. Google index

I will need to look into this to confirm but I think Docusaurus is generating fresh maps with every release.

3. Missing LINK

The LINK element is optional. It can be left out of the source code without any issue, pages will still render as expected.

4. Missing SYNOPSIS

Synopsis should never be missing IMO.

6. Links to wiki

Links still pointing to the deprecated wiki should definitely be replaced with links pointing to the pester.dev website

7. Test for help

This sounds like a great idea to me.

8. Multiple links

Get-help expects one single link/reference per .LINK entry. Stick to that convention and the correlation web pages will be rendered correctly. I would personally insert a whiteline separator in the example you gave above.

fflaten commented 4 years ago

Not sure if this belongs here, but might add the about_Pester help file is very outdated (v3?). E.g. it mentions New-Fixture which is not available atm in Pester v5 (found while looking at #1723 )

fflaten commented 3 years ago

How can we best document the assertion operators? The private functions like Should-BeOfType are usually documented, but they are hidden behind the public Should which only include a few examples. Moving/copying all examples to Should would make that page huge. Tips? Document them as static content and add links in Should?

Did the move to Docusaurus.Powershell stop the plan for markdown-first help, https://github.com/pester/Pester/issues/720#issuecomment-457475603?

bravo-kernel commented 3 years ago

@fflaten I'm not unsure I understand what that markdown-first comment means but IMO Docusaurus.Powershell is markdown-first. You can put any markdown element like links, bold, italic, etc. in the Get-Help and Docusaurus.Powershell will render it as markdown on the function-page.

fflaten commented 3 years ago

I read the mentioned issue and comment as a plan to replace the current comment based help with external markdown files as source and use PlatyPS to create MAML xml help for the module.

asears commented 3 years ago

Could includes of .ps1 files be used with examples or linked to using Docusaurus?

Would it make sense to reference longer examples in the Pester/docs/Examples folder as another canonical example like planets?

It could be useful if the examples were analyzed, testable and visible outside or via links from the documentation, perhaps as a separately cloneable playground or community script repo. This probably just involves some parsing of the synopsis header and extracting to files?

Will see if there's other ways to expose the nuggets in help of private functions and comments in code.

I found this which may help provide some ideas. https://get-powershellblog.blogspot.com/2017/05/the-classy-platyps-automated-class-enum.html

and this https://github.com/facebook/docusaurus/issues/599

bravo-kernel commented 3 years ago

@asears could you please open a new ticket for this one and then add a concrete example of what you are aiming to achieve so I can understand better?

fflaten commented 5 months ago

Closing as most items are fixed and the issues has grown too large. Opening new issues for any remaining work.

Could includes of .ps1 files be used with examples or linked to using Docusaurus?

Would it make sense to reference longer examples in the Pester/docs/Examples folder as another canonical example like planets?

I believe this is covered by pester/docs#268. We already use mdx partials to reuse samples but it could easily be a docusaurus component referencing files when a larger example project is created. 🙂