Requirements for mocking

[adbertram]

Is there any way to invoke tests with mocks for functions that aren't available? I'm not too familiar on how mocking works under the covers. Is it a hard requirement that a system MUST have the same modules available on the system that's running the tests?

The reason I ask is because I occasionally get these errors: CommandNotFoundException: Could not find Command and it'd be nice to not have this dependency if at all possible.

[nohwnd]

No there is not, and for a good reason. Say you are writing a function that automatically deletes all users from AD that haven't logged in for more than a year. In an ideal world, where everything focuses on being testable, it would be really easy to do that. Microsoft would ship version of AD that runs in memory and you could spawn hundreds of instances, one for each of your tests. (Which is what they do with ASP.NET WebApi, for example.) This would make your tests independent of each other, and hence easy to run in parallel. It would also make them deterministic, and very close to an "ideal test".

In real world this is not the case, the AD needs to be installed on a server, and you can't just spawn hundreds of those in reasonable time (even though with the advances in native Windows Docker containers this is probably more realistic than I am trying to make you believe :) ). So you turn to mocking.

Mocking is a convenience thing that gives you the luxury of not having to setup all the infrastructure but at a cost. The cost is that you no longer work with the real dependencies. Those real dependencies have their shape (their interfaces / APIs) and their behavior (e.g. what they return as a response when you add a new user to AD). The mocks have only the shape (the same API), but the behavior gone, and it is up to you to make your mocks behave in a way that is as close as possible to the real dependencies, but without needing the infrastructure in place.

If you allow mocks non-existing functions, you do not have the shape nor the behavior, you have nothing. It is then much harder to make your code eventually work with the real dependencies.

I had a similar discussion with Brian Farnhill which he summed up into a blog post.

There are also other upsides the current approach has, you can for example use default mocks as guards to prevent calling destructive cmdlets (thing Remove-ADUser). When you make a typo in the guard you need to be notified about that. You can also filter mock usages based on the arguments that were passed in.

In general nothing is preventing you from "mocking" whatever function/cmdlet you want, you just need to define a function of the same name in a scope that is closer to the caller than the original function. In the linked blog post you can see that the recommended aproach is to install the cmdlets if you can, and if you can't then generate function stubs that will become the base for your Mocks. I would suggest that you throw from those stubs to prevent you from getting false-positive tests.

[johlju]

For more reference. In PSDSC xSQLServer resource we needed to be able to mock the cmdlets in the unit tests on development workstation without the need to install the actual modules. All contributors might not have that option to install the module. So stubs for the modules was created.

https://github.com/PowerShell/xSQLServer/blob/dev/Tests/Unit/Stubs/SQLPSStub.psm1

And the stubs module are created using a helper function (from Brian's work in SharePointDsc).

https://github.com/PowerShell/xSQLServer/blob/dev/Tests/Unit/Stubs/Write-ModuleStubFile.ps1

[adbertram]

That's a good idea creating stubs like that.

[nohwnd]

@johlju @BrianFarnhill Do you know if anyone else is using this stubbing for their modules? It would be nice to see how many adaptations they needed to do to make it work and then possibly make it into a generic function and make it part of Pester or at least document it on wiki.

[BrianFarnhill]

@nohwnd I know I've passed the advice on a lot, but I'm not sure of too many others that have actually needed it yet. I know in a few places they are just downloading modules from the powershell gallery, so there are only scenarios that don't publish modules there (or other nuget feeds) that need to be covered this way.

[nohwnd]

@BrianFarnhill Okay, then I will just add it to the list of things to blog about :)

[johlju]

@nohwnd I haven't seen it used much either. Might be that most run the tests on a machine that has the modules already present, or download it as @BrianFarnhill says, and don't know about this method. I too started running tests on the actually server having, in my case, SQL Server. Until I needed to run the tests in a CI that does not have any modules present. I then found Brians method of making stubs. The benefit of it was that I could code tests on any machine I'd like. The only thing I did to Brian's code (what I remember) was to make it render the stubs according to the style guidelines of DSC Resource Kit, and adding a throw so if the stub was wrongly used it would show in the tests.

If you blog about this that would be great. :)

[hbuckle]

Out of interest I pointed the script at one of the AzureRm modules - it mostly worked but it messed up a bit on cmdlets with dynamic parameters. It would certainly be nice functionality to have included in Pester.

[nohwnd]

@johlju Thanks for the info. I remember that Brian had to do quite a few changes to my stubbing function prototype to make it work for Sharepoint. Which lead me to believe that providing a generic function is not as easy.

But looking at your and Brian's code, it seems that the only problem are strongly typed parameters in Mocks, which is something that might be relaxed and we discussed it here but no work has been done towards that goal and to be honest I am not even sure if that is the right way to go. Maybe replacing the types with [object] is a better way to go. I will have to think about it.

Throwing from the stubs is definitely a good idea.

[johlju]

@nohwnd In the case of SQL Server, then SMO classes (Microsoft.SqlServer.Management.Smo) and other .Net classes (Microsoft.SqlServer.*) is installed as a separate package (or when SQL Server is installed). So to actually use the module stubs, the Net. classes they needed to be replaced by [Object] (so we again don't have to install anything to write tests). So if the machine actually has the classes then there is no need to replace them with object.

My thought was it would be cool if one could automatically build stubs of all the .Net classes too (with methods etc). Now we do have stubs of the classes, but done manually, to which we add logic. This is when the classes with methods are used in code, but also so we can enhance the tests, mostly to make sure what we get in the object is what we expected to get. So if these stubs was generated (with empty methods etc.) that would save some time. I was hoping New-MockObject would get us there, but did not take us all the way.

Example of stubs for Microsoft.SqlServer.Management.Smo; https://github.com/PowerShell/xSQLServer/blob/dev/Tests/Unit/Stubs/SMO.cs

We are lacking cmdlets for SQL Server, that is why the need to use the .Net classes directly in code. In the tests we load the SMO stubs with Add-Type. But to use these, we have to make sure to never have the real SMO loaded in memory.

[it-praktyk]

@adbertram, is it your question answered?

[JustinGrote]

Related to this thread, for simplicity in your test its pretty easy to define the command as a function right before mocking it. In my case with the Azure Powershell Functions they use a Push-OutputBinding function that doesn't exist in the local environment, so in my pester test definition I just did

function Push-OutputBinding([String]$Name,[Object]$Value,[Switch]$Clobber) {}

And then mocked it on the very next line. Works fine.

[jazzdelightsme]

I came intending to file an Issue requesting the ability to define a stub and a function in one shot.

# I don't like explaining to people why we have to have this next line:
function Something { throw 'will be mocked' }
Mock Something { 'hi' }

But then I read the arguments in this Issue, and they make a lot of sense. However, I have an additional authoring scenario that means I hit this problem a lot: nested functions. Consider:

# Script under test:
function Foo
{
    [CmdletBinding()]
    param()

    begin
    {
        function Something { 'original something' }
    }

    process
    {
        Something
    }
}

This test code will not work:

BeforeAll {
    . $PSCommandPath.Replace('.Tests.ps1', '.ps1')

    Mock Something { 'mocked' }
}

Describe 'Foo' {
    It 'does something' {

        Foo | Should -Be 'mocked'
    }
}

(And to clarify "will not work": the problem is that the test blows up with a CommandNotFoundException when trying to mock Something.)

Instead, we have to write this:

BeforeAll {
    . $PSCommandPath.Replace('.Tests.ps1', '.ps1')

    function Something { throw 'will be mocked' } # <-- this line is annoying
    Mock Something { 'mocked' }
}

Describe 'Foo' {
    It 'does something' {

        Foo | Should -Be 'mocked'
    }
}

What to do? In general, I agree with your arguments about not wanting to mock non-existent functions... but this scenario is also annoying, because our test code is littered with function thing { throw 'will be mocked' } with the mock then defined on the next line... and everybody asks "why do we have to have this redundant stub function?"

function thing { throw 'will be mocked' }
Mock thing { 'sigh' }

[JustinGrote]

@jazzdelightsme this module helps cut down on the boilerplate of it: https://github.com/indented-automation/Indented.StubCommand

[jazzdelightsme]

Thanks @JustinGrote! Looks like a neat module, which makes it easy to define stub commands for certain cases... but I did not see anything applicable to dealing with nested functions--did I miss it?

[JustinGrote]

@jazzdelightsme no but using nested functions in Powershell is generally considered bad Ju-ju, there's very little benefit to doing it and a whole host of testing and encapsulation/scope problems creep up. Any reason why you have to have nested functions?

[jazzdelightsme]

@JustinGrote I often use nested functions simply as a scoping mechanism: i.e. no other code outside of the outer function needs access to this inner function, so don't expose it. Like local variables, but for code. Or like a private mini-module. I have never heard of nested functions being considered "bad". It is unfortunate that it presents challenges for testing.