Add support for running Pester tests

rkeithhill commented 8 years ago

I'm not exactly sure how to do this but I think it would be nice to have. Maybe this needs to wait for VSCode to have test-runner support (which may never happen)?

daviwil commented 8 years ago

I think @dfinke has done this before in VS Code. For now it just requires defining a task which can run Pester.

To take it further, we could implement our own Pester harness (possibly in PowerShell Editor Services) which could integrate with VS Code in a couple interesting ways:

Use VS Code's text editor "decoration" support to add some highlighting or gutter icons to show which tests have passed or failed. One good example of doing this is Wallaby.js. Here are some API docs for text decorations.
Use VS Code's "Code Lens" support to mark Pester methods with a "Run test" command so that we could invoke test methods directly. Here are some API docs on the Code Lens support. Code Lenses are being used in Visual Studio for stuff like this:

Could be an interesting feature to add!

rkeithhill commented 8 years ago

This has been partially addressed by PR #97, #98 and #105. However what we want is for VSCode to support creation of the tasks.json file with the Pester test task runner already configured in it rather than require the user to do that. See https://github.com/Microsoft/vscode/issues/2263

TylerLeonhardt commented 6 years ago

Looks like support for creation of the tasks.json file is supported. All that's left is to have the PowerShell extension create a test task that runs Pester.

fourpastmidnight commented 5 years ago

There's a great extension called Test Explorer UI, which is sort of a general "framework" for showing tests for a project. For each supported language, an adapter extension must be created, of which there currently are several: Jasmine Test Explorer, Mocha Test Explorer, .NET Core Test Explorer, just to name a few.

I had a brief thought to create such an adapter for Pester specifically (I can't remember the Pester issue I commented on about this very thing.)

TylerLeonhardt commented 5 years ago

@fourpastmidnight that would be awesome and would be a great addition to our ecosystem :)

Someone recently did something similar with Code Coverage https://www.pwsh.site/powershell/2019/01/10/how-to-enable-coverage-markings-in-vscode-for-your-powershell-projects.html

applying Pester to general purpose extensions.

It's a great idea!

SydneyhSmith commented 5 years ago

https://code.visualstudio.com/api/references/contribution-points#contributes.taskDefinitions this link shows how to define your own custom tasks

JustinGrote commented 4 years ago

@TylerLeonhardt this one may be closeable, I don't know if there's anything that needs to be done in the extension itself vs. testing-specific extensions or tasks, not sure if it's really in the scope of this extension specifically.

rkeithhill commented 4 years ago

The intent with this one was to have a command to add a "run Pesters tests" task to the workspace's tasks.json file. While being able to use the code lens feature is nice, if you have a lot of script files it isn't so convenient to run all the Pester tests in the workspace. Hence, I have a task in my projects to run all the project's Pester tests.

JustinGrote commented 4 years ago

@rkeithhill You can now have "global" tasks in your user settings, I have this in mine:

    "launch": {
        "compounds": [],
        "configurations": [
            {
                "name": "PowerShell: Interactive",
                "request": "launch",
                "type": "PowerShell"
            },
            {
                "args": [
                    "${command:SpecifyScriptArgs}"
                ],
                "cwd": "${file}",
                "name": "PowerShell: Current File",
                "request": "launch",
                "script": "${file}",
                "type": "PowerShell"
            },
            {
                "args": [
                    "${command:SpecifyScriptArgs}"
                ],
                "name": "PowerShell: Pester",
                "request": "launch",
                "script": "Invoke-Pester",
                "type": "PowerShell"
            },
            {
                "name": "PowerShell: Attach",
                "processId": "${command:PickPSHostProcess}",
                "request": "attach",
                "runspaceId": 1,
                "type": "PowerShell"
            }
        ]
    },

Is documentation guidance sufficient enough or should the is be an actual commmand palette item to populate something like this in either the user or workspace settings (user choice)?

rkeithhill commented 4 years ago

I think the generation of this "tasks goo" by this extension would be nice to have. I'm pretty sure this can be done as of tasks 2.0.0.

fourpastmidnight commented 4 years ago

I would note that for gherkin style Pester tests that I have a PR (#1276) that will soon be merged to the Pester project. I think I may have mentioned this about a year ago. Things have progressed more slowly than I had hoped—but any day now, it should finally be merged. These style tests will have new problem matchers for them.

In this PR, I also have a tasks.json file that contains many different tasks for running Pester tests, both Rspec style and Gherkin style—I personally use these tasks for running the Pester test suite.

Sent from my Windows 10 phone

From: Keith Hill Sent: Friday, April 17, 2020 11:43 To: PowerShell/vscode-powershell Cc: Craig E. Shea; Mention Subject: Re: [PowerShell/vscode-powershell] Add support for running Pestertests (#87)

I think the generation of this "tasks goo" by this extension would be nice to have. I'm pretty sure this can be done as of tasks 2.0.0. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

JustinGrote commented 4 years ago

@fourpastmidnight Also, things like this can be added using Set-EditorCommand in modules themselves, maybe it makes sense for Pester to provide a contribution to add the command to auto-populate the task rather than having it be inside the extension?

The only drawback that I see there is that it is less discoverable, and also precludes the use of having a toast-style popup "You don't currently have a Pester task defined. Would you like to set up a task for pester?"

fourpastmidnight commented 4 years ago

Ah, I didn’t know about this. I’ll have to read up on it and see what can be done!

Sent from my Windows 10 phone

From: Justin Grote Sent: Friday, April 17, 2020 13:32 To: PowerShell/vscode-powershell Cc: Craig E. Shea; Mention Subject: Re: [PowerShell/vscode-powershell] Add support for running Pestertests (#87)

@fourpastmidnight Also, things like this can be added using Set-EditorCommand in modules themselves, maybe it makes sense for Pester to provide a contribution to add the command to auto-populate the task rather than having it be inside the extension? The only drawback that I see there is that it is less discoverable, and also precludes the use of having a toast-style popup "You don't currently have a Pester task defined. Would you like to set up a task for pester?" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

TylerLeonhardt commented 4 years ago

All,

Upvote this :) https://github.com/hbenl/vscode-test-explorer/issues/112

(or ya know... Implement one)

JustinGrote commented 4 years ago

Pester can output NUnit, maybe it can be shimmed somewhat into this extension? https://marketplace.visualstudio.com/items?itemName=wghats.vscode-nxunit-test-adapter

fourpastmidnight commented 4 years ago

At some point, I also plan on allowing the Gherkin runner to output Cucumber Json format for told such as Donut.

Sent from my Windows 10 phone

From: Justin Grote Sent: Friday, May 29, 2020 10:50 To: PowerShell/vscode-powershell Cc: Craig E. Shea; Mention Subject: Re: [PowerShell/vscode-powershell] Add support for running Pestertests (#87)

Pester can output NUnit, maybe it can be shimmed somewhat into this extension? https://marketplace.visualstudio.com/items?itemName=wghats.vscode-nxunit-test-adapter — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Tiberriver256 commented 3 years ago

Is this still a requirement now that Tyler made this? https://marketplace.visualstudio.com/items?itemName=TylerLeonhardt.vscode-pester-test-adapter

JustinGrote commented 3 years ago

VSCode is getting a native testing API, maybe it should be open or renamed to target implementation/port of Tylers adapter/etc. to that, not sure? https://code.visualstudio.com/api/extension-guides/testing#test-results

TylerLeonhardt commented 3 years ago

I think generally, this issue could be closed because at this rate, I don't know if the PowerShell team will have the bandwidth to reimplement/maintain my extension.

Fair enough in my book considering the PowerShell team doesn't maintain Pester itself.

I'm happy with my extension being THE Pester extension...although I could always use assistance with maintaining it 😅

With all that said, I think we should sort out a few things before closing this.

Short term:

we need a way to disable just the Pester CodeLens' provided by the PowerShell extension so that folks can only see the CodeLens from my extension. Today, you end up seeing 5 CodeLens (2 from PS ext, 3 from mine) which is silly and confusing.

Long term:

a proper upgrade scenario for discovering my extension. That could include disabling the CodeLens if my extension is installed, and if not installed, and the user is using Pester 5, maybe notify them to check out my extention.

None of this work is hard, it's just not a priority which I get.

TylerLeonhardt commented 3 years ago

@JustinGrote the person who did Test Explorer UI (what my extension uses today) is working with my team on a migration scenario.

Once this PR goes in, you'll be able to switch to the new experience without me having to do anything 😁

https://github.com/hbenl/vscode-test-explorer/pull/199

andyleejordan commented 3 years ago

@TylerLeonhardt Now that the PR you mentioned went in, can we mark this issue as resolved?

JustinGrote commented 3 years ago

This can be closed, I agree with Tyler that even though Pester is considered a "de-facto" part of the ecosystem, it's not something the team supports directly and doesn't need to be owned by the Powershell Team.

I've been working on a reimplementation that uses the test APIs natively as an excuse to learn typescript and if it ever gets there I will release it as a standalone option. I don't see any reason this would need to be part of vscode-powershell directly unless it needed to leverage the same PSES instance for whatever reason, but instead probably better the vscode-powershell expose that as an extension API export instead.

JustinGrote commented 3 years ago

Although related: vscode-powershell provides the codelens for Run Test and Debug Test, and does have a Pester Stub. Should plans be made to deprecate and sunset that functionality?

TylerLeonhardt commented 3 years ago

There at least needs to be a setting to disable it so I can automatically apply the setting and migrate customers

JustinGrote commented 3 years ago

Agreed, I'd like to be able to have a popup that says "do you mind if we disable the native Pester codelens?"

If this gets marked up-for-grabs I'll consider doing it if mine ever reaches a ready-to-go state.

TylerLeonhardt commented 3 years ago

opened https://github.com/PowerShell/vscode-powershell/issues/3430

PowerShell / vscode-powershell

Add support for running Pester tests #87