Closed nohwnd closed 4 years ago
We had a discussion, and the consensus was that everybody prefers to go with the new features, even though the migration will be more difficult. And that Pester v4 will only receive patches.
There were multiple questions:
That would be everything connected to the discovery, better filtering, the new result object, and audit etc.
PowerShell 2 support in Pester is slowing the development down and making contributions difficult because not everyone has experience with it. So it's about time.
It was discussed and I think it will be a killer feature to have, especially for environment testing. So it will be in v5.
I might have missed some of the questions, I can't remember more than those.
There was also the question about speed. Would keeping Pester 4 compatibility make Pester 5 faster or slower. The answer was that at this point in time, Pester 5 is a little slower, especially around Mocks, but it's early implementation. There is still room for improvement in terms of speed. Converting the core functionality into a binary module to speed things up is also a possibility under consideration.
I'm concerned about Gherkin support 😉 . I'm not saying that I don't want to move forward with v5 and the breaking changes that entails. I'm in favor of the breaking changes. But, a lot of these new features are being developed without any regard to Gherkin (as I perceive things, and, if my perception is correct, understandably so).
It begs the question: what is the future of Gherkin within the Pester framework? Through my usage of the Gherkin runner, I have found that, while the concepts between Gherkin-style and RSpec-style tests are similar, they are not similar enough to share the same code. In v4, this has manifested when trying to use mocks, and they just don't work the same as they do for Pester proper. I've had to write a lot of custom code (which works well, but it was a lot of work) to work around the issue. It's not a fix, it's merely a workaround. Also, the way in which Gherkin-style tests are executed is inefficient compared to Pester. (That may be helped by some v5 features--but I also have a branch/PR that is attempting to resolve this issue specifically for Gherkin).
But, will the new features being developed for Pester-proper work for Gherkin-style tests? Or are these two testing styles different enough (as I suspect they are) that, realistically, Gherkin should be maintained almost as a separate "sub-framework" within Pester? (If I had a "Magic Button", the Pester proper would be a PowerShell test runner/executor, and there would be an RSpec and Gherkin "plug-ins" that the runner would use to execute the tests.)
@fourpastmidnight I don't know much about Gherkin. But right now there are the internals and the rspec layer over it, the mocking has separate implementation as well where it is separated into two parts so I am thinking about Gherkin but I am not doing any implementations towards it right now as I want to get the rspec version shipped first. As I said few times, once v5 is out let's talk about this, then I will have a good idea of what is possible and how to use it for gherkin 🙂
I understand the desire to drop PowerShell v2 support in Pester v5 to aid future development and agree with that approach. However, I would suggest not capping PowerShell v2 support at any point in Pester v4. It is so great to have testing available for PowerShell v2, as I still come across old dusty customer servers running v2.
As far as I understand it, you don't plan new features for Pester v4, so I would hope that the pain of supporting PowerShell v2 should be minimal and restricted to bug fixes. Apologies if I'm off the mark.
@TigerBoom that sounds reasonable to me. :)
What about PowerShell v7 support in Pester v5?
@tmack8080 did not try it yet, need to research if we can already build it somewhere on one of the build servers or if I need to provision a vm for it. I guess it helps that PowerShell itself is using Pester to test it, so if it won't work we hopefully will hear about it from the PowerShell team.
@tmack8080 asked it on twitter and people successfully used Pester v4 on PowerShell 7. The goal is to make Pester available on any version and any platform so its users can test all their code no matter where it runs.
Is there "Whats new in Pester 5.0" docs? Or all news present in the issue?
@iSazonov see this readme: https://github.com/pester/Pester/blob/v5.0/README.md
The content will move to release notes after release and the old readme will come back. Also there is more info in the releases already: https://github.com/pester/Pester/releases
TLDR;
Personally I would love to do a lot of breaking changes in the release of v5, because the advantages of the new features are far outweighting the downsides. But I understand that people have a sizeable test suites that would be very difficult to migrate to v5, so a side-by-side support for v4 would be needed.
Which route to choose:
New features
Dropping PowerShell v2 support
The supported versions of PowerShell will change from PowerShell 2+ to PowerShell 3+ (including 3) in Pester v5. Dropping support for v2 is long overdue, and it does not make sense to keep dragging this burden into the next version of Pester. Pester v4 still supports v2, but I plan to cap it at version 4.9.x, and make future updates to Pester v4 only support PowerShell 3+.
Test discovery
Discovering which tests are in a given
*.Tests.ps1
file was something that I wanted Pester to do since I joined the project some 5 years ago. And now it can finally do it. The price to pay for it is resonably small, you need to put all your code into Pester controlled blocks. On the other hand you got a lot of benefits:Better result object
The whole internal architecture is now centered around a hierarachical result object, that looks something like this, but with way more information, and more nested:
Such structure reflects the actual layout of your test file much better, because there is one level per describe, and allows for much easier and more flexible post processing. For example a file with:
Would produce a result object that contains three levels, 1st level describing the file itsels, 2nd level describing Describe d1, and 3rd level describing Describe d2.
So using
$result.Blocks[0].Blocks[0].Tests
shows metadata of test i1, including if it passed, the error record associated with it, the time it took to run, and so on.Operational state is stored in the result object
This tree structure is also used by Pester to store it's internal state, so the state that used to be runtime-only is stored in here as well. After execution you can inspect the object and see for example which mocks were defined in this test, how many calls were made to that mock or other. This brings a much better post-mortem capabilities because storing this result object will allow you to inspect the run without debugging it live.
Corrected scoping of Describe
Test such as this will pass in Pester v4, because
BeforeAll
is invoked before theDescribe
in which it is defined. This means that the variable$a
will be available for the whole subsequent script, leading to unexpected results, because the script scope is polluted by the variables.Schematically it looks like this,
.
means import into the current scope and&
run in new scope:In Pester v5, the scoping is corrected. One more scope is added above Describe so d1 BeforeAll is isolated and does not leak variables into the script, and subsequest Describe.
Schematically it looks like this, notice that each describe is contained in another scope:
Corrected scoping of It
In Pester v4 the
BeforeEach
is executed in a scope aboveIt
, so variables defined inBeforeEach
are not writable fromIt
and not changed inAfterEach
, this makes sharing data between the test and the teardown difficult. For example when you created a file in the test and want to remove it, or close a connection to a database etc.Notice that the value of variable
$be
is not changed.Schematically it works like this, the one extra scope around
It
is causing the problem:In Pester v5
BeforeEach
It
andAfterEach
all run in the same scope, notice that$be
changed to2
:Schematically:
Overall the idea here is that, the two following should be equivalent, so if multiple tests have the same setup or teardown you can extract the setup or teardown to BeforeEach / AfterEach without changing the behavior:
Should be equivalent to:
Fixed scoping of Mock
In Pester v4 the mock defined in the
It
block is defined on the containingDescribe
, so it leaks to the subsequestIt
blocks. This is a hurdle that everyone learning Pester faces. It also forces the test author to add unnecessaryContext
blocks when the mock should be "isolated" to oneIt
block.In Pester v5 the Mock is defined on the block that contains it. So putting it within
It
will define this mock for thatIt
and putting it intoBeforeAll
will define it for theDescribe/Context
that contains it. Leading to a more obvious behavior.Notice that the function
f
is mocked only in the firstIt
and the functiong
is mocked in both because the mock is defined on theDescribe
level.Mock assertion name change
Should
is the way to do assertions, yet in Pester v4 there are two notably different assertions,Assert-MockCalled
andAssert-VerifiableMock
, in Pester v5 these commands are deprecated andShould -Invoke
andShould -InvokeVerifiable
are newly available, and recommended to use.Proposal: Compound assertions as default & default PassThru on assertions
Writing tests with multiple assertions is not-recommended because it gives you just partial information on failure. The way to do this correctly is to write multiple
It
blocks, each having just a single assertion. Or we could allow assertions to tell the framework that there was an error, but not actually fail. This combined with outputting the asserted object, and maybe some extra assertions, would allow for chained assertions that only fail when the test is done, not at the first failed assertion:So here the first assertion would behave like the normal Pester assertion, it would fail the whole test, because it uses
-ErrorAction Stop
. The next two assertions would inform the framework that there is a new error and the test would fail in the end, but both of the assertions would run, not just the first one to fail.I think this leads to a very readable tests, and would be a huge improvement over the current approach.
🙋 Add other stuff, suggest your own.