dingmeng-xue
commented
4 years ago @rodolfograve , what's the expected result when using WhatIf? I didn't figure out the meaning of -Whatif to Connect-AzAccount.
Closed rodolfograve closed 4 years ago
dingmeng-xue
commented
4 years ago @rodolfograve , what's the expected result when using WhatIf? I didn't figure out the meaning of -Whatif to Connect-AzAccount.
rodolfograve
commented
4 years ago Hi, and thanks for the response.
WhatIf is used very frequently for testing purposes, as a way to make the commands not to perform any action that would require a real resource to be affected in any way.
I agree that in the case of Connect-AzAccount it's not obvious what WhatIf should do if you look at it from the "hypothetical" excecution point of view. However, it's easier to define if we come to it from the "how would you test a script that makes use of Connect-AzAccount" angle.
I suggest something like:
"Connect-AzAccount -WhatIf" should return a fake PSAzureProfile object, either with well-known, hard-coded, fake Tenant and Subscription ids, or using random values.
In terms of code, my suggestion is that we add an else here: https://github.com/Azure/azure-powershell/blob/f1b754320ca279d65949ae18740b20440b773c89/src/Accounts/Accounts/Account/ConnectAzureRmAccount.cs#L329-L338
This is pseudo-code. I haven't looked at the classes involved to see if they have a constructor we could use, etc. but it should be possible to make those modifications:
else
{
WriteObject(PSAzureProfile.CreateFake());
}Another option is that I'm just wrong trying to use -WhatIf for this purpose, but I haven't found any other way to write tests for PowerShell scripts that target resources. Any suggestions that work around this issue are very welcome.
dingmeng-xue
commented
4 years ago @rodolfograve , many az powershell cmdlets support -WhatIf but they need the correct context and some operation needs real HTTP call to check status. If we return faked context, I believe other cmdlets with -WhatIfwill be failed. Could you share more why you cannot do Connect-AzAccount in test env?
@mikefrobbins , @dcaro , @markcowl for your awareness
rodolfograve
commented
4 years ago @dingmeng-xue , thanks again.
I think there are two separate issues/topics here:
I think #1 is a fundamental issue. As the author of a script that uses a script that uses a script (...) that uses Connect-AzAccount, I don't expect things to fail only because I added the WhatIf flag. Everyone expects WhatIf to change the behaviour of the script so that it doesn't perform any real operations, but not that it changes other aspects of the integration of the script with other scripts.
If possible, I would like to focus on that aspect as the most important.
As for #2, the main reason is that we are trying to avoid having credentials embedded in the code. This creates a security problem, but more importantly, it makes your tests fails if the "right" environment is not available:
In an attempt to illustrate this one (examples are always dangerous), imagine I want to test the Do-Something command below. Please, keep in mind that in practice the scripts are much more complicated and nested:
Function Connect-ToMySubscription {
[CmdletBinding(SupportsShouldProcess, ConfirmImpact="Low")]
Param()
$Credentials = Get-MyCredentialsFromEnvironment # So that the script connects to the right subscription according to the environment
Connect-AzAccount -Credentials $Credentials -Tenant "MyProductionTenant" -Subscription "MyProductionSubscription" -ServicePrincipal
}
Function Get-Something {
[CmdletBinding(SupportsShouldProcess, ConfirmImpact="Low")]
Param()
Connect-ToMySubscription
# Execute some Get- Az commands
$Result = Get-....
return $Result
}I think issue #1 is well evidenced here. Get-Something would return different things with -WhatIf than without.
And it would be a shame to water down our very secure approach to managing secrets only because we are not able to test the scripts :-)
rodolfograve
commented
4 years ago Hi @dingmeng-xue . Do you have any updates on this? Maybe @mikefrobbins , @dcaro , @markcowl?
dingmeng-xue
commented
4 years ago I'm revisiting -Whatif concept and I think Azure PowerShell is using the conventional way.
-Whatif displays a message that describes the effect of the command, instead of executing the command.
PS C:\Users\dixue> $a = New-Item -ItemType File -Path .\newfile1.txt -WhatIf
What if: Performing the operation "Create File" on target "Destination: C:\Users\dixue\newfile1.txt".
PS C:\Users\dixue> $a
PS C:\Users\dixue> Connect-AzAccount -WhatIf
What if: Performing the operation "log in" on target "User account in environment 'AzureCloud'".Thanks @dingmeng-xue . That definition of -WhatIf does not mention anything about returning values and what happens when scripts interact using -WhatIf.
I think I gave a couple of arguments in my previous comment. Is there any chance the Azure PowerShell team can take a look and address those arguments?
Test-ability is a very important feature of a framework. If you see Azure PowerShell as something people use interactively then you won't care too much, but there is a huge potential for people to write complicated scripts and even entire modules that build on top of Azure PowerShell (as we are!), and the current behaviour for -WhatIf is a deal breaker for that scenario.
dingmeng-xue
commented
4 years ago @rodolfograve , I understand your point from test perspective. But I'm still confused about expected behavior. Do you have any example from PowerShell core modules for my reference?
Below I tried New-AzResourceGroup and New-Item. Neither has returning value.
PS C:\> $a = New-AzResourceGroup -Name abc -Location westus -WhatIf
What if: Performing the operation "Replacing resource group ..." on target "abc".
PS C:\> $a | fl
PS C:\> $b = New-Item -Path .\a.json -WhatIf
What if: Performing the operation "Create File" on target "Destination: C:\a.json".
PS C:\> $b | fl@dingmeng-xue apologies for the delay, I was disconnected for a few days.
I can't find any examples of commands that return a value when -WhatIf, so it looks like I'm making an unfair request to the PowerShell Az team.
That does not mean the existing pattern is right, tough. I can't think of any other way in which we can test our scripts.
At this point all I can say is that I would appreciate any guidance on:
Thanks a lot for all your help so far.
dingmeng-xue
commented
4 years ago Sorry that we have no test approach using a mock up Connect-AzAccount. In fact, we have 2 approaches to test Azure resources.
Connect-AzAccount . Here is brief introduction https://github.com/Azure/azure-powershell/blob/master/documentation/testing-docs/using-azure-test-framework.md.I cannot figure out other approaches. If you know any way in practice, please let me know.
dingmeng-xue
commented
4 years ago Close this issue now. Please contact us if you need further help.
Description
The Connect-AzAccount function writes an object to the standard output when not using -WhatIf. However, it doesn't when -WhatIf is present.
Since WhatIf is key when writing tests, this breaks tests for commands that use Connect-AzAccount and return a value of its own. This is because of PowerShell's behaviour to treat any standard output as a value returned by the function.
I believe this is the section causing the issue: https://github.com/Azure/azure-powershell/blob/f1b754320ca279d65949ae18740b20440b773c89/src/Accounts/Accounts/Account/ConnectAzureRmAccount.cs#L329-L338
WriteObject(...) is inside the if (ShouldProcess...) and there is no else to return some fake value.
Writing tests for PowerShell scripts is already hard enough, particularly for those scripts that target some infrastructure (most of them?). Hopefully key frameworks like PowerShell Az can relieve some of the pain by providing first-class support for testing.
To be clear, the issue here is that my tests would pass (they use WhatIf) but my scripts will fail when used for real.
Steps to reproduce
Environment data
Module versions
Debug output
Error output