Future of Gherkin in Pester

[nohwnd]

Originally posted here: https://github.com/pester/Pester/pull/1276#issuecomment-610318255

Right now Pester v5 tries to be accomodating to be layered with another framework "face", and while it was useful to think about the internals as being modular to keep the internal architecture clean, it brings a lot of overhead, and unnecessary renaming, just to support <1% of users.

I am thinking about removing it entirely from v5, and splitting it from v4 to it's own repo under pester organization where hopefully @fourpastmidnight and you could rule and improve it as you see fit? Either forking v5, or maybe I could still keep the "runtime" part distinct from RSPec Pester and rename the items during build and not during runtime. And you'd do the same to build Gherkin Pester. It would then ship in a nuget under a different name, say PesterGherkin.

original response from @fourpastmidnight in the same thread

@nohwnd Re: future of Gherkin in Pester.

I have mixed feelings about this. I've actually wanted to approach you and ask for your thoughts on this very subject. Let's start a new issue to pick up this conversation. I feel like it's one worth having if only to "clear the air" and properly set the stage for not only the future of Gherkin, but possibly even Pester itself. 😄 I think some of what you mentioned makes a lot of sense. But like I said, I do have mixed feelings. So let's do a Cost-Benefit analysis in another thread. (I don't want to speak for @renehernandez, but he did comment on this PR; so perhaps he should be an initial participant, too?)

@Jaykul @renehernandez move the dicussion here if you are interested.

[Jaykul]

As background, my initial reason for writing gherkin as a "flavor" for Pester rather than a separate test module was primarily so I could reuse core functionality instead of rewriting it:

1. Mocks
2. Code Coverage
3. Output (particularly converters to NUnit xml and code coverage xml)

Of course, it also gave me a way to get gherkin "in the box" so that a certain class of user doesn't complain about modules using Gherkin instead of Pester.

Finally, I hoped to persuade certain people that RSpec was awful and not inherent to Pester, and that Pester should create it's own first-class DSL

[fourpastmidnight]

First off, THANK YOU @Jaykul for writing this initial implementation of a Gherkin test runner utilizing the Pester framework. I was so glad to have literally stumbled across this about 18 months ago!

While I share many of @JayKul's sentiments, I wonder if having an embedded Gherkin implementation will really change people's minds at this point? Or, I wonder how many people just don't know that a Gherkin runner even exists within Pester? When I first started looking for a PowerShell test runner, all the YouTube videos and blogs featured the RSpec runner only. (Maybe because when that content was created the Gherkin runner didn't exist yet? And/or when they started creating their test suites, they had started with RSpec because Gherkin hadn't existed yet.) Also, I recognize that writing Gherkin style tests is quite different from writing other unit tests--even RSpec ones. There's more indirection with Gherkin. I think this puts some people off, rightly or wrongly.

And in v4 of Pester, mocking using the Gherkin runner is tricky as all heck and doesn't work quite the same way as it does with RSpec. When I say tricky, I mean that certain things just don't work with Gherkin tests as they do with RSpec tests (I'm looking at you InModuleScope)—maybe I just don't know how to do it properly; and if that's the case, better documentation is needed. And when looking online for help, invariably, all you see are examples for RSpec. I wonder if this alone has led to less usage of Gherkin over all?

Anyway, I agree that the biggest advantage of having Gherkin be a part of Pester is being able to reuse all of what you said: Mocks and Code Coverage. Though, with #1276, output has largely been re-written from scratch. That being said, IF the Gherkin runner were to be split from Pester proper, I would love to see an individual Pester.Core module that contains the core Mocking logic which can be reused by (potentially) other runners. And as @nohwnd said, perhaps a Pester.RSpec module and a Pester.Gherkin (or even Pester.Cucumber) module which contains the two supported Pester test runners.

This may, in fact, be beneficial for the project at large. As the core is changed, the runners can change somewhat independently of the core. Realistically, in order to verify the core is working as intended, I suppose one or the other runners would need to be updated to use the new features. But I digress. Another area where this could be beneficial is that it would allow for test runner implementations to better support distinct features. For example, a common request in this repo is to support more and more output formats, both test and code-coverage. The runner modules could decide which formats they'd like to support. (Though, there was a GHI topic discussion on here about modularizing Pester for that kind of thing. I don't think we ever really got anywhere on that issue. It's worth revisiting. It's also relevant to writing console output, too—is there a way to generalize that, perhaps provide a "base" implementation, while allowing for specialized implementations? This gets back to @Jaykul's point about why he implemented the Gherkin runner the way he did: to reuse as much as possible to get something working as soon as possible with the least amount of effort possible.)

The one issue with splitting this off is one or the other runners (and most likely Gherkin more than RSpec) being a bit "behind the curve" when it comes to updates to the Pester Core. But, I don't see this as a HUGE issue, as that's already happening with v5 of Pester. 😞 (This is the nature of open source, though. I do this in my free time, as do most (if not all) contributors to this project.)

All that said, I think that splitting Gherkin from Pester could work. But I would equally like to see RSpec split from Pester, too.

(As an aside, we would just create a dependency between the RSpec/Gherkin modules on the Pester.Core module, no? So is the reusability issue really an issue?)

[nohwnd]

Thanks @Jaykul the good thing is that Output, Mock and Code Coverage are now way more independent than they were before, they are all a plugin that is registered with the core. You can quite easily (I hope) consume Mock and CC, and use your own Output plugin (they are just a function that create an object that has a bunch of scriptblocks (callbacks)). I would just have to generalize the RSpec part of Mock (it figures out the mocks defined based on the -Scope that differentiates between Describe and Context, even though the runtime part only considers them all to be blocks), or you would have to write your own if you need a different behavior.

function New-PluginObject {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [String] $Name,
        [Hashtable] $Configuration,
        [ScriptBlock] $Start,
        [ScriptBlock] $DiscoveryStart,
        [ScriptBlock] $ContainerDiscoveryStart,
        [ScriptBlock] $BlockDiscoveryStart,
        [ScriptBlock] $TestDiscoveryStart,

    # ... etc.

You say "Pester should create it's own first-class DSL", and I agree, I would love that. And you can see it in my naming in the pester.runtime, I use New-Test, New-Block, New-Mock etc... That is how Pester would look like if it would be written for powershell. This naming then leaks into the result object, and so there are multiple names for the same thing. I don't think this is nice, and I don't think there is much room for the PowerShell syntax Pester (but I asked here https://twitter.com/nohwnd/status/1247752097677709312?s=20 ). So currently I am on a fence whether I should rename all of the stuff in the result object to __Tests__ so I can replace all references to it automatically with It. Naming and framework specific choice parameters are the "difficult" part here. I can alias New-Block to Describe, but people will want to see Describing abc not Block abc in the output. And I can make it happen. But if there is no drive to have powershell syntax Pester then why bother.

I see how difficult it is to drive people to use something else that does not come in the box with my Assert module, that I use a lot, but people always wonder if it is usable for Pester, and why it is not built in.

Anyways back to the original topic:

The downsides I see:

• gherkin ships dlls that most of the people don't need, making the shipment unnecessarilly big
• it publishes cmdlets that almost nobody needs
• in v5 I do a lot of work to keep the internals "generic" to be able to accommodate another test framework with it's own behavior, and that makes the whole thing less performant than it could be
• gherkin is not understood by me and so it is hard to maintain it, but luckily @fourpastmidnight does a great job at that
• gherkin is not understood by me and so I don't feel confident shipping it

[smokedlinq]

I have used both and see value in both. I use gherkin tests about a third of the time, mainly depending on the complexity of the solution, I feel more business driven specs work well in gwt style when a larger solution is built than single function driven tests in rspec for simple scripts.

I consider the mocking capabilities one of the core capabilities of Pester. I feel if some how they are separate that from that then they can be freed to define tests and scopes as each frameworks sees fit to remain valuable and evolve independently.

Using multiple modules wouldn’t be the worst thing. I wonder what other dsl testing frameworks (thinking operational validation/integration) could come from such a design.

Just my two cents.

[nohwnd]

@fourpastmidnight

Or, I wonder how many people just don't know that a Gherkin runner even exists within Pester?

I think many are aware, but not many tried it. As you say there is barely any content about it, because it did not reach the "critical mass". It was exactly like this with Pester as a whole when I joined the project. There were probably 4 people in the world using it. So I started using it and writing about it. It got some traction, Dave joined, and Kevin picked it up and made videos about it, then Adam and Don, and it slowly got more and more traction. It was probably the right time for it as well. I remember giving a talk about using git in your CI pipeline at Dutch powershell group few years ago, and barely anyone there used git. Now everyone uses git, ci and pester.

There's more indirection with Gherkin. I think this puts some people off, rightly or wrongly.

This was always the problem for me, I never felt like I want to deal with writing the unit test, and deal with parsing as well, when just writing the unit test gets the job done.

I would love to see an individual Pester.Core module that contains the core Mocking logic which can be reused by (potentially) other runners.

Probably a separate repo would be easier, or folder in Pester repo that you can pull from (to make my life easier). I would still like to ship the "main" pester as Pester.

There is such module in v5 it is called Pester.Runtime, and it is sorrounded by CC plugin and Mock plugin.

As the core is changed, the runners can change somewhat independently of the core. Realistically, in order to verify the core is working as intended, I suppose one or the other runners would need to be updated to use the new features.

This already happens in v5, the rspec indedpendent parts are largely tested by invoking them directly. https://github.com/pester/Pester/blob/v5.0/new-runtimepoc/Pester.Runtime.ts.ps1 In many places I simply rely on the existing RSpec tests, but the core is first ensured to be working by tests that target just it. This allows the low level functionality to be validated with relative ease, and in isolation.

For example, a common request in this repo is to support more and more output formats, both test and code-coverage.

In v5 internals are driven towards the runner executing the tests and then piping into a cmdlet that provides you with transformation to the desired output from the output object. This is a bit muddled in the current release because I heavily filter the result object, see this thread for what is shown now vs what is actually there (and it only show the container level) https://github.com/pester/Pester/issues/1480

Then for convenience the Invoke-Pester function bundles some of the functionality so you can say Invoke-Pester -CI and get coverage.xml, testresults.xml and exit on error, but internally it passes the result object to the respective functions. So once full api is published (and full result object) you should be also able to replicate -CI with (pseudocode):

$r = Invoke-Pester -CodeCoverage $ccConfig -PassThru -EnableExit
$r | Export-JacocoReport -Path coverage.xml
$r | Export-NUnitReport -Path testresults.xml

In the current implementation the runtime decides when stuff should run, and how it is scoped to keep the scoping consistent, and then plugins do their own stuff when they are called, for example CC work in a way that in Start plugin callback it instruments the files with breakpoints, stores them in PluginData.CodeCoverage on the result object, and returns.

function Get-CoveragePlugin {
    New-PluginObject -Name "Coverage" -Start {
        param($Context)
        $config = $Context.Configuration['Coverage']
        $breakpoints = Enter-CoverageAnalysis -CodeCoverage $config
        $Context.GlobalPluginData.Add('Coverage', @{
            CommandCoverage = $breakpoints
            CoverageReport = $null
        })
    } -End {
        param($Context)

        if (-not $Context.TestRun.PluginData.ContainsKey("Coverage")) {
            return
        }

        $coverageData = $Context.TestRun.PluginData.Coverage
        $breakpoints = $coverageData.CommandCoverage

        Exit-CoverageAnalysis -CommandCoverage $breakpoints
    }
}

On End it removes the breakpoints and returns. Before this Invoke-Pester figures out which files should be covered, and after this Invoke-Pester pipes the result into the export function.

It works similarly for mock and output in their Mock plugin and Write-ScreenPlugin.

https://github.com/pester/Pester/blob/3635ca0fcbde37153ddbc73ecbed7f1b46870113/Functions/Output.ps1#L443

Mock is admittedly a bit more complicated because the RSpec way of -Scope might be different from Gherkin idea of how Mocks should be scoped so that implementation is separate, but the core logic of setting a mock up or counting the hits from a list of provided mocks is common. And if our scoping ideas agree the non-common implementation can also be largely re-used.

The runner modules could decide which formats they'd like to support. (Though, there was a GHI topic discussion on here about modularizing Pester for that kind of thing. I don't think we ever really got anywhere on that issue. It's worth revisiting. It's also relevant to writing console output, too—is there a way to generalize that, perhaps provide a "base" implementation, while allowing for specialized implementations?

"behind the curve" when it comes to updates to the Pester Core. Yes that is already happening and migh happen in the future, but it also has an upside of me not having to make sure that adding a new feature to the core will not break gherkin, because you will have your own schedule of updating the core.

(This is the nature of open source, though. I do this in my free time, as do most (if not all) contributors to this project.)

Totally true, sitting here every morning from 7-9 the past few weeks, and every weekend for the past year and more to make v5 happen. 👍

All that said, I think that splitting Gherkin from Pester could work. But I would equally like to see RSpec split from Pester, too.

On technical level it already did I believe. There is Pester, there is Pester.Runtime and there would be Pester.Gherkin. I won't rename Pester to Pester.Rspec if you meant that 🙂

(As an aside, we would just create a dependency between the RSpec/Gherkin modules on the Pester.Core module, no? So is the reusability issue really an issue?)

This is actually the biggest part of the problem. The runtime creates a tree object that contains all the data. While this is great thing in most cases (and I still pat myself on the back occasionally for going that way 😁 it has one minor downside, the naming you choose on the object will leak into the result that the user of the respective flavor sees. This can be easily fixed by renaming the properties, but that has a significant impact on the performance, because you need to recurse the whole object and add a new property for the one that you want to rename, and remove the old one. This takes significant portion of the time, and worse is not associated with any "work" that the user would see. From the user perspective it just hangs there processing the result after testing is done.

Also the cmdlets that process the result need to understand the naming and how to navigate the tree say $result.Containers[0].Blocks[0].Tests[0] would look nicer as $result.Files[0].Describes[0].Its[0] and would use the same names everywhere in RSpec Pester. Not to mention that Blocks would correctly translate to Describes and Contexts but I am not willing to do that because the benefits are tiny, instead I would call everything Describes and give it a Type that would say Describe / Context to avoid complicating the block structure and navigation and all code.

I could put rel like link on each object .NextBlocks = "Describes" .NextTests = "Its" on each object, but I think it is simpler and cleaner to just rename all such core specific stuff to Tests and Blocks so you can do -replace "Blocks", "Describes" when you build the core for your own usage.

Ultimately having an internal module is slow, calling functions is slow and loading files from more than one file is also slow. So Pester will end up putting everything into 1 file with 0 internal modules and probably inlining some helper functions like any. That is once I write a build script to do this, and in there renaming "Blocks" would happen.

[nohwnd]

@smokedlinq Thanks for the feed back, very valuable.

Using multiple modules wouldn’t be the worst thing. I wonder what other dsl testing frameworks (thinking operational validation/integration) could come from such a design.

I doubt there would be many, but I might be surprised. While you can write a basic testing framework in under 100 lines of code. The work associated with writing a test framework that actually works, supporting and improving it is significant. The main complexity lies in mocking and scoping. And while the core already handles the heavy lifting for you, the RSpec mock adapter which only figures out what mocks were defined in the above scopes based on the -Scope parameter, and does some other minor stuff, is still ~600 lines of code that took me a month of evenings to get right.

Granted I was also rewriting the internals at the same time, but I don't think many people will be willing to write something that is significantly different from Pester, instead of just layering it on top of Pester and have me maintain it and improve it.

[fourpastmidnight]

If the decision is to more strategically separate the runners from the "core", perhaps we could use Git submodules? I hate git submodules--but maybe that could work here? Anyone else feel strongly one way or another about that possibility? It may help solve the problems with using different PowerShell modules that @nohwnd mentioned. It's not a perfect solution and introduces some complexity of its own, but it provides a bit more isolation than just another folder in the repo.

I'm thinking that by having submodules, each of these "components" of Pester will then have their own repo and associated history which wouldn't "pollute" the history of the core pieces of Pester, but they're still "tied together" with the main Pester Core repo.

[nohwnd]

I also dislike submodules. I kinda like the way vscode-powershell does it, they locate powershelleditor services clone next to their clone. Imho it is simple. Needs no git magic and keeps all repos easy to manage separately, and would be easy to communicate in a single error message.

[Jaykul]

I don't have any issue with having a separate module. I have completely stopped caring about anyone who won't use things that aren't "in the box", so the upsides/downsides kind of work out now.

With the old code, everything was tied to the $Pester scope object, so to the extent that's no longer necessary, and the functions can be used from outside, Gherkin could take a dependency on Pester and not care too much about the extra download -- or, it could submodule Pester, and only use the parts of the source that it needs ... whatever. 😉

I'd be willing to help with adapting it to 5, although I think I'd be tempted to wait until the performance has caught up a little. In the context of Gherkin, I'm not sure what the benefit of updating is.

[nohwnd]

The poll has shown that pursuing PowerShell native syntax does not make sense, people like that it is different from their production code, that they can use their existing experience to write powershell tests, and that the syntax is easy to grasp to new-comers. So I will go forward with the ___ renaming that I mentioned above so I can adapt the result object to my liking on build time, and so can you. And I will see what to do with functions, because now there is a lot of indirection, and calling functions is expensive, so reducing the amount layering that happens is necessary. For better perf.

https://twitter.com/nohwnd/status/1247752097677709312?s=20

[fourpastmidnight]

I'm not sure I understand.... This poll seems to indicate that ppl would prefer New-Test, New-Block, New-Setup, etc. (If I understand this poll correctly, I would've answered No, I like the RSpec style over the PowerShell syntax, and obviously I prefer Gherkin over RSpec 😉 )

In any event, I think you're saying that you feel Pester should not use the PowerShell syntax? That RSpec style would prevail? Am I correct?

[nohwnd]

What I meant is that the poll is not 90% for powershell syntax and 10% for rspec. So there is no point in changing what works, and adding the burden of yet another syntax on myself. 🙂

[nohwnd]

Anyways I will start cutting Gherkin from Pester 5, and start renaming to __ where necessary.

[fourpastmidnight]

Sounds good! Would it be better to merge #1276 first, or are you feeling that this should wait until you've made the split?

[nohwnd]

You are merging into master which has v4, that will not affect me in any way. I think it can be merged as soon as I get to it :)

[ianwalkeruk]

Wow, thanks for the big lead time. You could have at least marked the gherkin functions as deprecated in the latest v4.

You don't even list this as a breaking change in the notes. Just bang - Pester v5 is released, and all the gherkin users can go swivel?

Not cool

[nohwnd]

Sorry, did not know that there are any / many users. There is community discussion thread since Jan 2019, https://github.com/pester/Pester/issues/1218 where there are no mentions of Gherkin, and few other threads like this one https://github.com/pester/Pester/issues/1245. The common theme there is that it is just me, Craig and Jaykul talking.

[nohwnd]

Added to the release notes. I hope we find community of several thousands people who are secretly using Pester Gherkin for years 🙂

[ianwalkeruk]

"Not all who wander are lost" - J.R.R. Tolkein "Not all users follow GitHub discussion threads" - Me

Sorry if I was a bit grumpy and rude earlier - I've just spent a couple of months convincing a team to build their testing around Pester-Gherkin!

I know form reading other comments you have doubts over the use case for it, but the BDD style works well for the corporate infrastructure/platform engineering space, because the separation of "what we want to test" from "how we test it" means I can take a feature file to a non-technical business analyst or product owner (or customer!) and we can talk about the "what" without the "how" getting in the way of that discussion.

Please if there's an effort to get a separate Pester.Gherkin module running against Pester.Runtime v5+, can I be included, and I'll lend my meagre programming skills when I can?

[nohwnd]

Sure, no problem, no offence taken. 🙂 You can totally continue using Pester-Gherkin on v4, and even help us testing the new improvements in https://github.com/pester/Pester/pull/1276 .

And I count you in into the new Pester.Gherkin 🙂

[sheldonhull]

Sorry, did not know that there are any / many users. There is community discussion thread since Jan 2019, #1218 where there are no mentions of Gherkin, and few other threads like this one #1245. The common theme there is that it is just me, Craig and Jaykul talking.

Hey, I love gherkin! I blogged about it recently too, loving the implementation! 👍 My biggest problem in using was I only really had one blog series that I used to learn about the details. Was not as easy to dive into as the normal pester tests. I look forward to seeing separate implementation. I think it's really neat for collaboration, but if I was teaching someone pester, I do think learning both Pester DSL and Gherkin runner syntax is too much, so I've reverted to mostly just native pester, also since I'm mostly writing these for myself I'm not gaining the benefits of using Gherkin in a collaborative environment.

[richardwadsworth]

✋ another Gherkin fanboy here. Was so happy to discover the Pester Gherkin implementation about 6 months ago. I'd be happy to contribute to the new Gherkin implementation.

[dennisl68-castra]

Anyways I will start cutting Gherkin from Pester 5, and start renaming to __ where necessary.

Any info about where the stand alone version of Gherkin for PowerShell can be found? I really need it for high level User Acceptance Test and User Acceptance Criteria descriptions/discussions as my users doesn't read Pester/PowerShell natively...

[nohwnd]

There was no progress on moving Gherkin to the new core that is used in Pester v5, and there was sadly no progress on my side on extracting the internals to make that easier either. At the moment there it Pester 4.10.2-beta1 that has the latest gherkin changes #1276. I am waiting for some time, and some beta testers to indicate it is okay (mainly on the RSpec side, the Gherkin side @fourpastmidnight has been using for the past two years).

[dennisl68-castra]

Ok, so Gherkin is still in active development in Pester v4.x? :)

[nohwnd]

Not in active development, but it's a stable version, that recently got a PR merged that was pending for two years.

[dennisl68-castra]

Ok, so quite a long test period then? ;)

[DennisL68]

So, have the tests been concluded yet?