In-memory selection of the current storage account isn't working properly

[KirkMunro]

Description

Related to #6054 and #6186, if you are using in-memory caching for your Azure PowerShell work, the Set-AzureRmCurrentStorageAccount cmdlet will run successfully, but then commands such as Get-AzureStorageContainer will fail because the storage context that they need is not available in the cache.

There are a few problems here:

1. Set-AzureRmCurrentStorageAccount returns a string instead of a storage context (What? How does that make sense?).

2. When you are using in-memory caching, the storage context appears to be stored in memory (the command runs without error at least), but then other commands such as Get-AzureStorageContainer fail to retrieve the storage context from memory. These other commands should be able to retrieve the storage context from memory in this case.

Module Version

6.2.1

Environment Data

Name	Value
PSVersion	5.1.17134.112
PSEdition	Desktop
PSCompatibleVersions	{1.0, 2.0, 3.0, 4.0...}
BuildVersion	10.0.17134.112
CLRVersion	4.0.30319.42000
WSManStackVersion	3.0
PSRemotingProtocolVersion	2.3
SerializationVersion	1.1.0.1

[maddieclayton]

@blueww Can you take a look at this issue?

[blueww]

@KirkMunro Would you please give more detail for "in-memory caching"? How do you configure it? It's better if you can provide a script that can repro the issue on a normal computer.

Besidest that, Set-AzureRmCurrentStorageAccount will create a Storage Context and save it in Default Context. Default Context will be used when the following Storage Data plane cmdlets (as Get-AzureStorageContainer) don't input storage context. Set-AzureRmCurrentStorageAccount will only output the storage account name as the context is already saved in Powershell Default context.

If you need a storage context as output, you can use

(Get-AzureRmStorageAccount -ResourceGroupName $rgname -StorageAccountName $accountName).Context

Thanks!

[KirkMunro]

@blueww Here's how I configure my PowerShell runspace for in-memory caching (note, I'm doing this in C#, but this is the PowerShell equivalent, and it assumes you have a few variables predefined, like $credential, $subscriptionId, $tenantId):

Disable-AzureRmContextAutoSave -Scope Process -ErrorAction Stop

Add-Type -LiteralPath 'C:\Program Files\WindowsPowerShell\Modules\AzureRM.Profile\5.0.0\Microsoft.Azure.Commands.Common.Authentication.Abstractions.dll'

$azureAccount = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureAccount]::new()
$azureAccount.Id = $credential.UserName
$azureAccount.Type = 'ServicePrincipal'

$azureSubscription = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureAccount]::new()
$azureSubscription.Id = $subscriptionId

$azureTenant = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureTenant]::new()
$azureTenant.Id = $tenantId;

$azureAccount.ExtendedProperties['Subscriptions'] = $azureSubscription.Id
$azureAccount.ExtendedProperties['Tenants'] = $azureTenant.Id

$azureSubscription.SetAccount($azureAccount.Id)
$azureSubscription.SetEnvironment('AzureCloud')
$azureSubscription.SetTenant($azureTenant.Id)

$azureTokenCache = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureTokenCache]::new()

$azureContext = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureContext]::new()
$azureContext.TokenCache = $azureTokenCache
$azureContext.Account = $azureAccount
$azureContext.Subscription = $azureSubscription

$defaultProfile = Connect-AzureRmAccount -DefaultProfile $azureContext -ServicePrincipal -TenantId $TenantId -SubscriptionId $subscriptionId -Credential $credential -ErrorAction Stop

$PSDefaultParameterValues['*:DefaultProfile'] = $defaultProfile

Once I have that set up, I run my other PowerShell commands. This is the context under which I think Set-AzureRmCurrentStorageAccount should work, and while it appears that it does, subsequent calls to commands like Get-AzureStorageContainer do not work. I'd like to know if this is simply a bug or an unsupported scenario.

Further, in environments where disk caching is meant to be completely disabled, any commands that would cache to disk should return an appropriate error indicating that disk caching is disabled, guiding users towards the proper commands that they should use.

Let me know if you have other questions.

[blueww]

@KirkMunro

Would you please give more background like why you create the $azureContext, instead use the default? With more background, we can better help you.

I have tried to run the script from you, but the following command fail.

$azureSubscription.SetAccount($azureAccount.Id)
$azureSubscription.SetEnvironment('AzureCloud')
$azureSubscription.SetTenant($azureTenant.Id)
$azureContext.Subscription = $azureSubscription

with error like:

Method invocation failed because [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureAccount] does not contain a method named 'SetTenant'.
At line:3 char:1
+ $azureSubscription.SetTenant($azureTenant.Id)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : MethodNotFound

Add I have installed Azure PowerShell 6.2.1, and it include AzureRM.Profile 5.2.0, instead of 5.0.0. So jsut to confirm you really use 6.2.1.

One question is what's the fucntionality of following command? $PSDefaultParameterValues['*:DefaultProfile'] = $defaultProfile

Anyway, I can repro the issue. It seems the resource mode profile don't have DefaultContext correctly set with this script. Since I don't own this part of code, I will need more time to investigate it.

[KirkMunro]

@blueww Thanks for coming back with questions.

First, regarding the script not working, I'm sorry about that. As I mentioned, I'm running this from C#, not from PowerShell, so all of the commands that set up the context are invoked from .NET. When I converted this yesterday, I forgot to check for extension methods, and I suspect the methods that you cannot invoke are extension methods, so I will see what needs to be done there to get that to work.

In regards to your question about background, I create the $azureContext because I am running scripts in Azure functions, which share the same appdomain/process, and Azure functions must be stateless, so I use in-memory caching instead of letting the default process/on-disk caching occur. That is why issue #6186 was created, so that anyone doing this wouldn't have to write all of this intermediate code to generate an in-memory context. For now though, there is no cmdlet to make this just work, so the extra context code (everything between the call to Disable-AzureRmContextAutoSave and Connect-AzureRmAccount) is required.

For your last question, setting $PSDefaultParameterValues['*:DefaultProfile'] allows me to ensure that I can continue to invoke AzureRM cmdlets without having to manually pass in the in-memory context as the default profile to each one of them. With this set, any AzureRM cmdlet that I invoke that has a -DefaultProfile parameter will automatically be passed the in-memory context that I create. This gives me an in-memory caching model that still allows me to invoke AzureRM cmdlets easily. Normally they just pull the default profile from the process cache (which uses static variables behind the scenes), which I don't have to work with. This model gives me runspace/thread-level caching, allowing each runspace to work independently of each other runspace that is run through an Azure function.

[KirkMunro]

@blueww Sure enough, the issues you ran into were because of extension methods. Sorry for not checking for those earlier. Here is an updated script that will properly invoke the extension methods that you pointed out could not be invoked properly in your last post:

Disable-AzureRmContextAutoSave -Scope Process -ErrorAction Stop

Add-Type -LiteralPath 'C:\Program Files\WindowsPowerShell\Modules\AzureRM.Profile\5.0.0\Microsoft.Azure.Commands.Common.Authentication.Abstractions.dll'

$azureAccount = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureAccount]::new()
$azureAccount.Id = $credential.UserName
$azureAccount.Type = 'ServicePrincipal'

$azureSubscription = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureSubscription]::new()
$azureSubscription.Id = $subscriptionId

$azureTenant = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureTenant]::new()
$azureTenant.Id = $tenantId;

$azureAccount.ExtendedProperties['Subscriptions'] = $azureSubscription.Id
$azureAccount.ExtendedProperties['Tenants'] = $azureTenant.Id

[Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureSubscriptionExtensions]::SetAccount($azureSubscription, $azureAccount.Id)
[Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureSubscriptionExtensions]::SetEnvironment($azureSubscription, 'AzureCloud')
[Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureSubscriptionExtensions]::SetTenant($azureSubscription, $azureTenant.Id)

$azureTokenCache = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureTokenCache]::new()

$azureContext = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureContext]::new()
$azureContext.TokenCache = $azureTokenCache
$azureContext.Account = $azureAccount
$azureContext.Subscription = $azureSubscription

$defaultProfile = Connect-AzureRmAccount -DefaultProfile $azureContext -ServicePrincipal -TenantId $TenantId -SubscriptionId $subscriptionId -Credential $credential -ErrorAction Stop

$PSDefaultParameterValues['*:DefaultProfile'] = $defaultProfile

[blueww]

@KirkMunro

If you run the Azure function from C#, why don't you just use the SDK directly instead of use PowerShell in C#, it will be more nature and flexable?

• You can use Management SDK to get account Key. How to use it.
• Then use Storage Client Library to create storage account object with the account name/key, then do blob operation.

Is there any special reason that you want to manage Azure through Powershell in C#?

Let me know if you have any difficult to use the SDKs.

BTW, It seems the $azureSubscription = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureAccount]::new() should be changed to $azureSubscription = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureSubscription]::new()

Could you confirm?

BTW, if you can share the C# code, it might will also be helpful for the investigation.

[KirkMunro]

@blueww

In reverse order:

1. I cannot share the C# code behind this. But it is doing the same as the PowerShell script I have provided. Set-AzureRmCurrentStorageAccount and Get-AzureStorageContainer get invoked after the script I have provided.

2. You are correct, that was a typo in my script, which I have corrected above. My apologies for the typo.

3. Of course I could use C# with the Azure SDK if all I needed to do was programmatically work with Azure. My requirements are broader than that, and PowerShell is a hard requirement on this.

[blueww]

@KirkMunro

I tried to debug the issue, and found it might be related with the command: $PSDefaultParameterValues['*:DefaultProfile'] = $defaultProfile

Set-AzureRmCurrentStorageAccount will set the storage account to the DefaultProfile.DefaultContext. (normally, it should be the RMprofile, but in this case, it should be the profile you set.) Get-AzureStorageContainer will get the storage account by sequence from RMprofile, or SMProfile, and enviromentVariable. But seems the dataplane cmdlet don't have access to the DefaultProfile.

@markcowl , @cormacpayne , do you have any idea on this problem? How can we resolve it?

@KirkMunro, could you give more details for why you must use C# and Powershell? I still don't get the reason.

[KirkMunro]

@blueww Reasons why I must use C# and PowerShell:

1. I need to capture results from the PowerShell that is invoked in an application. Using C# to invoke the PowerShell script makes this very easy.
2. I define additional binary cmdlets that are loaded automatically into the PowerShell session where the PowerShell script runs, so that those cmdlets can be invoked from within the PowerShell script.

Those are two key reasons why I'm doing it this way, but there are more. Doing it entirely in PowerShell is not an option, nor is doing it entirely in C#.

[blueww]

This issue is caused by the Storage dataplane cmdlets don't support paramter "-DefaultProfile", so can't get the default stroage context in a user input profile. I have tried to add the "-DefaultProfile" paramter to dataplan cmdlets, and try to get storage context from default profile (sequence: RMprofile, SMProfile, DefaultProfile and enviromentVariable). And the issue not repro with the change. (See the temp fix in https://github.com/wastoresh/azure-powershell/commit/b6d7234826d84c185fcb196f59f45e07563be52b)

@markcowl , @cormacpayne I am not sure if this kind of fix has any potencial issue, since Profile is not owned by our team. Do you have any comments for the fix?

[blueww]

@KirkMunro

The fix is merged and should be in the next Azure PowerShell official release (should be 6.6.0) Please check when it's released.