Gherkin...

[Jaykul]

Would you guys like gherkin support? I've got this working on top of pester:

http://poshcode.com/paste/445/

Disclaimer 1: I just started playing with the idea yesterday.

Disclaimer 2: It uses the official gherking parser -- which depends on a large pile of DLLs (the IKVM package)/

I have obviously not accounted for everything yet , and I'm still trying to figure out your best practices for loading/unloading modules etc, but so far it works for me ;-)

[nohwnd]

Looks interesting, I need to have a detailed look tomorrow on what gherkin is. :)

[Jaykul]

Isn't that how "everyone" does BDD nowadays? ;-) They call it "specification by example" ...

It's the specification "DSL" that was created by Cucumber, the docs for it start on the cucumber wiki

But it's also used by Python behave, .Net SpecFlow, Selenium, and others.

[dlwyatt]

Looks pretty cool! Is this a modification of Pester, or a separate module that uses Pester as-is to do some of the work?

[Jaykul]

It could be either, but it would be a lot easier for if it became part of Pester, because Pester doesn't export a lot of it's functions -- stuff like Should are not available from outside.

Obviously it's a very different style of writing tests (and I haven't finished implementing tables and examples) and even the reports are a bit different, so I do realize it could get confusing to document both of methods together, but this style is what I've used at all my jobs since BDD came into favor:

Gherkin gives me management-readable spec files (we train the PMs to write them, and work with them to edit if need be), which they can sign off on as an acceptable specification, and then we implement tests directly on that, and can report directly to that as well.

Of course, I still use more traditional unit tests when bugs pop up that weren't just weaknesses of the specification ;-)

Technically speaking:

I wrote Invoke-Gherkin as an alternative for Invoke-Pester (which parses .feature and .step.ps1 files instead of .test.ps1)

I wrote When as a replacement for It ... although it works very differently, technically, because I need to register when the script is initially parsed, rather than executing (each "when" function could be run many times based on the feature file), and I had a hard time getting the tests to run inside the Pester scope (I'm not 100% sure that I've got all the perfect yet).

The actual test code is basically the same as what you'd write in any Pester test, with parametrization wherever that's useful. That means I'm still relying on all your state, mocking, assertions (and code coverage?).

To ship outside I would have to inject into Pester, which might work, and might not. In the worst case, I'd be telling people to install Gherkin "over the top of" Pester (which would basically be like forking Pester just to add my files and change two lines on the psm1).

[nohwnd]

Are you using Pester 2 (master branch) or Pester 3 (Beta branch)? Because the version 3 exports Should and all the other Pester keywords (it has to, because the code no longer runs inside of the module scope). Just a little heads up, so you don't have to rewrite all your code when we release Pester 3 (in upcoming weeks???).

For me the optimal solution would be identifying the metaphors of the two grammars and implementing a testing core that would abstract from particular keywords. Not sure where the intersection is, though, as I know pretty much nothing about cucumber/gherkin at the moment. But what I saw in the examples the ? "mapping" could go like this:

Describe -> Feature
Conxtext -> Scenario
It -> Given, When, Then

What would And and Or map to? Or am I totally wrong?

[dlwyatt]

Yep, basing this on the Beta branch (v3.0) would make your life a lot easier. Here are the exported functions as of now. Everything else is internal, and Pester takes care of executing script blocks in the correct scope.

\Documents\GitHub\pester [Beta]> Get-Command -Module Pester

CommandType     Name                                               ModuleName
-----------     ----                                               ----------
Function        AfterEach                                          Pester
Function        Assert-MockCalled                                  Pester
Function        Assert-VerifiableMocks                             Pester
Function        BeforeEach                                         Pester
Function        Context                                            Pester
Function        Describe                                           Pester
Function        Get-TestDriveItem                                  Pester
Function        In                                                 Pester
Function        InModuleScope                                      Pester
Function        Invoke-Mock                                        Pester
Function        Invoke-Pester                                      Pester
Function        It                                                 Pester
Function        Mock                                               Pester
Function        New-Fixture                                        Pester
Function        Setup                                              Pester
Function        Should                                             Pester

Invoke-Mock isn't intended for use from a test script, but it still needs to be exported in order for module mocking to work. I suppose I should add something to that effect in comment-based help, in case someone comes along looking at this same output and wonders what it's for.

[dlwyatt]

We should probably just lay out the pros and cons of having the modules separate or combined, and make a decision from there. Here's what I can think of, off the top of my head:

If the modules are combined:

• Easier implementation; the Gherkin-specific code would have access to all of Pester's internals, which might make it work more smoothly. (At the cost of making it somewhat harder to change these internal functions, since they'd become a dependency for two different parts of the module.)
• Possibly enhanced functionality. Rather than a separate command, it could just become part of Invoke-Pester which could then process both traditional *.Tests.ps1 files as well as Gherkin .feature files. This could make CI solutions more straightforward, as you'd wind up with a single summary of test results / single NUnit xml file / etc.

If they're separate modules:

• Users wouldn't have to download all of the Gherkin parser DLL bits if they don't intend to use that functionality.
• Gherkin support is forced to use Pester's public API, which makes it easier to make changes to the rest of Pester's code without breaking Gherkin. (Though we could enforce this internally, even if the modules are combined.)
• It would be easier to test the Gherkin adapter code if it's in a module separate from Pester. Because of the separation of module scope (different SessionStates), you could mock calls to Pester commands like It or Should in a unit test of the Gherkin code. Mocking those in Pester's own scope would probably blow up the tests.

[Jaykul]

Yeah, that's pretty much correct, @nohwnd. I did switch to the Beta yesterday, and if "Should" is out, this Gherkin core probably could work separately, and you have the mapping correct as well.

All of those keywords: Given, When, Then, And, But .... are the same. They exists because they make the specifications easier to map to natural language. You can use any of them in the language, but at execution they all map to "When" -- the design of gherkin is that you can reuse phrases in multiple tests, so my phrase: Given a module with the name "<name>" can exist in another scenario as And a module with the name "Bob" ... and you'll see in the one that I have now the phrase "Then the module's local storage path should be '...'" and "And the folder '...' should exist" which get repeated in every test I have written so far ;-)

The key functionality of "Gherkin" is to allow you to write natural spoken language specifications that get turned into tests. The biggest benefit, therefore, is that you get complete specs. ;-) When a "bug" is found that wasn't specified, I can clearly say to the stakeholders: look you didn't have that in your specification.

You can obviously see that Given-When-Then basically maps directly to Arrange-Act-Assert, (Given->Arrange), (When->Act), (Then->Assert) ... which makes me ask: where did the Describe-Context-It grammar come from?

When I wrote my personal testing framework "PESTER" (I'd point you to the blog post, but it's missing from the internet right now) I based it on arrange-act-assert and mapped those to PowerShell's begin/process/end keywords ... My core idea was to turn TEST: Arrange-Act-Assert into a DSL representation of FUNCTION: Begin-Process-End ... this allows piping in data so that you could do the same test with multiple sets of data. The problem was that writing the tests that way was uncomfortable, and since only the process block in run repeatedly, it ended up being awkward and forcing parts of your arrange and assert steps into the act.

Generally as I write my steps, I use "When" as the keyword for everything (or sometimes the "Given" alias, because I can't help myself), but the key difference between Gherkin and Pester's traditional system is that my "When" keyword just declares the test, exactly like the function keyword, and then execution of the test happens as we parse the configuration, and each step function may get executed many times.

[Jaykul]

P.S. One of the things that seem missing to me in Pester (and this may just be because I haven't explored enough, or found up-to-date docs) is that ability to run a test multiple times with different inputs and expectations. I just added support for "Examples" to my Gherkin invoker (see that paste again), and it's pretty cool (right now I only support it on the Scenario, and it should work on individual steps too).

So for instance, in Gherkin, if you have a "Scenario Outline" instead of a Scenario, then you can use and an example table, and each step gets matched and executed once for each row of the table. You could write parametrized tests, or you could write a test for each match ... it's up to the person writing the tests.

As a side note, I found it odd yesterday to find that the parser supports multiple named example tables --- I can't figure out why you would do that...

[dlwyatt]

Describe/Context/It are part of the language used by RSpec, which I assume was the inspiration for Pester's current language.

[nohwnd]

@Jaykul thanks for the summary, I need to read the Cucumber book, or at least the relevant parts of it to better understand what you are saying. Gonna take few days :)

To run the same test multiple times with different input you can place the relevant part of the test inside a function or a loop. There is no special support for this.

[Jaykul]

@dlwyatt ok, let's see. Assuming that with the scope changes Gherkin can execute outside of Pester and just take a dependency, I think it boils down to this: If you want to add Gherkin to Pester, then you should add me to the team (personally, I think that's a :+1:) and we would keep a single product branding :+1: -- we would have some C# code and some dlls from a nuget package -- so we might end up wanting two packages, but we get me to work together within Pester and thus avoid any risk (however small) that I would end up forking Pester.

In more detail:

Regardless of which we do

• Having Gherkin support will help convince developers who are used to using Gherkin that this is the right test framework for them (i.e.: brings more users)
• Having Jaykul writing Gherkin code based on on Pester means that he's finally using Pester instead of his own thing, and "we" can count on having occasional features and bug fixes (and long threads like this one).
• Gherkin is awesome :100: and eventually everyone's going to use that syntax ;-)

BUT

• Pester probably needs a simple unit-test style command (and examples) like "Test"
• Pester should define it's core grammar better in documentation, currently you talk about Arrange-Act-Assert but you don't do it, instead you have this Describe-Context-It and you need to explain (or link to an explanation of) WHY you're using these weird words (a mention of and link to RSpec would be good)

Adding Gherkin to Pester

• Allows the Pester team some influence over how it's implemented
• Could result in beneficial refactoring which might mean even more ways of executing (for example, regular pester could get "test plans" as an "ordered" alternative to tags)
• Would allow a single execution function to run both specification .feature files and bug/regression/unit tests
• Could lead to tie ins, like using Pester to run Gherkin+Selenium tests

BUT

• Would mean committing to keeping both sets of functionality working
• Includes some C# code for the gherkin parser and shipping a dll
• Risks Jaykul leaving you with a pile of code to maintain that we don't use ourselves
• Might mean shipping two modules (especially if users object to the dlls)

Keeping Gherkin Separate

• Avoids maintenance of Gherkin for Pester team
• Keeps Pester simpler, allowing you to focus on one feature set
• Keeps Pester download smaller

BUT

• Dilutes brand (people may think they're using "PowerCuke" or whatever instead of "Pester")
• Risks confusing end users (note this is less of a problem once we get "OneGet" for PowerShell modules)
• Risks confusing codebase or even forking

[Jaykul]

P.S.: Oooooh, yeah ... RSpec ... I never could get a handle on that one. It makes no sense to me to write "specs" that don't specify the expectations (except in code form). Maybe that explains my initial reactions to Pester. Anyway, that's clearly a whole different thread ;)

[dlwyatt]

Pester should define it's core grammar better in documentation, currently you talk about Arrange-Act-Assert but you don't do it, instead you have this Describe-Context-It and you need to explain (or link to an explanation of) WHY you're using these weird words (a mention of and link to RSpec would be good)

Can't really help ya there; the Pester DSL is pretty much as I found it. My work on the project has mostly been about how, not why. :)

[nohwnd]

@Jaykul I am really glad extending Pester is worth your time, I read a bit about Gherkin and it looks like a great thing to have. But at the moment we first need to release the 3.0 version of Pester and document it a bit. Then we can start abstracting the core from what is specific to Pester. This should result in building blocks of a testing framework on top of which we could build Gherkin language support, Pester language support and maybe even support other PowerShell testing frameworks. Depending on how complicated adding a new language support will be we might split this into separate repositories, or not. Keeping the same name would be nice though.

Until all this happens, please keep your module separate, and if there is anything you need to patch in Pester please send a PR.

Btw: Would you care to share your code with us? :)

[dlwyatt]

@nohwnd, I can see some of the code in https://github.com/Jaykul/Pester/tree/Beta . Based on what I see there, I think the idea of having Gherkin code in a separate module for now probably won't work as well. He's not calling Pester's Describe, Context, or It functions, but rather copying some parts of their implementation and discarding others.

So we may be down to options of either merging the Gherkinized version of Pester into the main repo, pretty much in its current form, or we wait to add Gherkin support until we've overhauled Pester's implementation a bit, as you described. (Turning the core Pester module into a framework that language-specific modules can leverage, or something along those lines.) Once that's done, each language module (including one for Pester's current DSL) can remain separate, without having to duplicate code or expose internal stuff.

@Jaykul, food for thought:

If you want to add Gherkin to Pester, then you should add me to the team

You're on the team already, just by starting this discussion and contributing code and ideas. In the work that's gone into v3.0, I've wound up contributing more lines of code than any other individual who's worked on the project so far, and almost all of those contributions took place without me having any special access to the main repository. (I was given Collaborator access about a week ago, but only because @nohwnd has been busy working on other things, and didn't want me to have to wait to merge in some loose ends on v3.0 before it gets released.)

[Jaykul]

Yeah, @dlwyatt I know how it is ;)

All my code is in my github fork (although for now I've been just been rebasing my changes onto beta and then --force pushing to that github repo as a way to keep all my changes together in one series of commits for the sake of a future pull request).

I'd like to be able to just call the base Pester functions, but they all output strings like "Context" and "Describe" which make no sense outside of RSpec. There are a few flow control problems too (I'm looping) but those can probably be fixed. I spent a little time last night updating my functions from the beta3 versions (even though it was already working) so if you looked at it today, you should be able to see the resemblance ;-)

Once 3.0 is out of beta, it would make sense to look at what would need to be done to properly refactor and to avoid the copy-paste refactoring I've been doing -- but I don't really want to add that work to what you're already doing.

The biggest difference, as it stands, is that my When function doesn't execute the code, but rather stores it for execution later (based on the .feature file). Given some time, I should write some actual Pester tests for it too, since right now I'm just testing it by running it against that test module i put on PoshCode.

Ultimately I'm not sure if we will want to abstract out functionality, or if just share the assertions, mocking and code coverage pieces and consider the rest of it the difference between two "test runner" systems. We shall see :)

[Jaykul]

Ok, so ... it's time to talk about gherkin ;-) Let me go ahead and send a pull request.

[dlwyatt]

Let's start up a new branch for this work; we're likely to have to make smaller independent bug fixes for v3 in master while we're in the middle of the changes described in this thread.

[dlwyatt]

I've created two new branches, DevelopmentV4 and Gherkin. Please open this PR into the Gherkin branch for now.

[Jaykul]

Bump

[dlwyatt]

Haven't forgotten. :) I'm working two jobs at the moment, and haven't had as much free time to spread around open-source projects recently.

[dlwyatt]

I should hopefully have some more time to dedicate to Pester soon. When that happens, I'll start working on getting the current RSpec-ish language and its associated console output decoupled from the rest of the code (and improving our test coverage along the way.) Once that's done, we should be able to add clean Gherkin integration pretty quickly.

[KirkMunro]

Curious (and too lazy to go check for myself), what is the current status of this right now? I'm still itching to start playing with Pester and incorporating it into my module development workflow, but only if it has a Gherkin syntax. :)

[dlwyatt]

Not much progress yet, but I'm starting to work on it. I have my evenings back, now that I've changed jobs, but I haven't really got back into the swing of things yet. Holiday vacation coming up, should be recharged in January. :)

[dlwyatt]

Note: You could always just start using Joel's Gherkin branch, if that's a requirement for you. The usage may change later on, but as far as I know, he had a working prototype a long time ago. It just involved some messy duplication inside the Pester module.

As mentioned earlier in this thread, some of our plans for v4 are to define a fairly generic internal API, and then have the test scripts themselves parsed and imported by separate modules that can be plugged into Pester. The first task that I decided to tackle before making those changes was to improve our test coverage, as the Describe, Context and It commands were previously very hard to test due to some shared global state in the module. That's coming along, and pretty soon it'll be time to start ripping things apart. :)

[Jaykul]

Yeah, the Gherkin branch here (or on my fork) is stable and working -- the merging changes should be all about avoiding duplication and making Pester innards more amenable to testing, alternate syntax, and such.

When all is said and done, there shouldn't be any breaking changes to how the gherkin works (and it would be really nice to have someone else put it through it's paces to prove I've got it right).

[KirkMunro]

Excellent, that will do nicely short term then. Thanks for the fast reply guys.

-----Original Message----- From: "Joel Bennett" notifications@github.com Sent: ‎2014-‎12-‎18 7:22 PM To: "pester/Pester" Pester@noreply.github.com Cc: "Kirk Munro" poshoholic@gmail.com Subject: Re: [Pester] Gherkin... (#173)

Yeah, the Gherkin branch here (or on my fork) is stable and working -- the merging changes should be all about avoiding duplication and making Pester innards more amenable to testing, alternate syntax, and such. When all is said and done, there shouldn't be any breaking changes to how the gherkin works (and it would be really nice to have someone else put it through it's paces to prove I've got it right). — Reply to this email directly or view it on GitHub.=

[Jaykul]

I've rebased my Gherkin tree (and moved it, just because): https://github.com/Jaykul/Pester/tree/Syntax/Gherkin

I had to make quite a few fixes to make it work on 3.3.8

For the record, my output functions are no longer compatible with Pester's because of your changes to have everything go through Write-Screen -- originally I had moved ALL Write-Host calls in all of Pester into Functions\Output.ps1 -- in this branch, I've left the Pester code alone (so it's still calling it's own functions), and I've only used the new functions in the Gherkin side (to avoid throwing out all of someone's refactoring).

The new Write-Screen is too limiting for me (and frankly, seems like an unnecessary abstraction). Specifically, removing the ability to write more than one color on a line means I can't lowlight the timestamps, and I prefer a few more color options because I have at least one extra level of headline output.

I need to add a line to read my color configuration from a psd1 configuration file (like what I'm doing with strings), and I think that's what Pester needs too (regardless of whether we re-integrate the output functions or not).

[KirkMunro]

Oh! I have a really nice solution to the Write-Screen limitation that I've been working on...I'll see if I can polish that up and ship it, because it would be ideal for this!

[dlwyatt]

Write-Screen's main responsibility is to encapsulate the -Quiet switch to control calls to Write-Host. We can refactor out the other color logic into a separate function, which can be used by Invoke-Gherkin or Invoke-Pester as needed.

[dlwyatt]

Also, don't worry about doing too much of that in your fork. We've already merged Gherkin (minus these output-related merge conflicts) in the LanguageDecoupling branch, and have done some other improvements to improve our test coverage in that same branch. That's a bit of a misnomer now that I've sort of given up on that decoupling idea for the moment, but it's still where we're doing the work of getting this ready to ship.

[Jaykul]

Yeah, I just needed Gherkin working (on 3.3.8), so I could do some other work :grinning:

You had also mentioned the new switches, and so I added those -- obviously -quiet isn't actually implemented, since I haven't reconciled Write-Screen, but I figured too much had changed in the output functions within master and LanguageDecoupling to be able to merge them back together, I guess I was assuming it was a dead end.

I could have a look at doing it the other way (merge master to LanguageDecoupling) if you think it's worth it -- but one way or another, we definitely want to have only one set of output functions (I currently have two because of Write-Screen).

[dlwyatt]

I haven't done work on LanguageDecoupling in a couple of months, so it's already due for another big merge from master. Will do that soon, and then figure out what to do with the output code.

In a perfect world, we'd get @nohwnd 's "color scheme" functionality working with Invoke-Gherkin as well (and add your nice multi-color lines to Invoke-Pester while we're at it), but I suppose it's not a huge problem if they look different.

[Jaykul]

OK. (just for the record). I owe you some tests for the invoke-gherkin function.

[Xoph]

Did this ever get implemented? I would have use for this as well.

[nohwnd]

the gherking functionality should be available in the Pester/DevelopmentV4 branch, a developement branch for Pester 4. That version works pretty fine, but it's your consideration if you should use it in production environment before it's fully released.

[Xoph]

Cool! Thank you. We write our requirements in Gherkin, and we have a proprietary tool that analyzes the Pester results and traces them back to the Gherkin requirements, so having the Gherkin output would make it a step easier.

[nohwnd]

Give it a try it and see if it will work for you. I guess Jaykul is the only real user of this at the moment (or one of a few), so if you have some more questions I suggest you start a new issue and ask there, hopefully he will respond.

[Jaykul]

Do you think Pester 4 will ever be released? ;-)

[nohwnd]

Let's do it next week. :)

[gaelcolas]

@Jaykul @nohwnd @dlwyatt Am I right thinking this issue can be closed as Gherkin support has been released in v4? (and thank you for that!)

[Jaykul]

I think these are "Pending" or something while v4 is in RC?