Azure Session Context getting incorrectly applied to multiple instances of Foreach-Object -Parallel

[dansos]

Hello, I am currently working on some Azure scripts where I need to perform a series of operations on each of the subscriptions under our tenant. I encountered a large problem when using Foreach-Object -Parallel. After some debugging I came up with this short script as a means of reproducing the issue.

$subscriptiondata = Get-AzSubscription
$subscriptiondata | Foreach-Object -parallel {
    $log = "Creating file for subscription $($_.name)..." | Out-File -FilePath` `"C:\temp\test\$($_.name).txt" -Force
    $log =  "Setting the session context to subscription $($_.name)..." | Out-File -FilePath` `"C:\temp\test\$($_.name).txt" -Force -Append
    Set-AzContext -Name $_.name -Subscription $_.id -Force
    $context = Get-AzContext
    $log = "The context has been set to $($context.Subscription.Name)" | Out-File -FilePath` `"C:\temp\test\$($_.name).txt" -Force -Append
    $Array = @(1..50)
    Foreach ($number in $array){
        $context = Get-AzContext
    $log = "The context has been set to $($context.Subscription.Name)" | Out-File -FilePath` `"C:\temp\test\$($_.name).txt" -Force -Append
    }
} -ThrottleLimit 10

(Apologies for the poor formatting of the script above - github keeps screwing with it)

The above script proves that for some reason, the Azure session context is not getting applied correctly to individual parallel sessions and seems to somehow be "leaking" from its original session to the others. Quite problematic, as it forces us abandon the parallel processing for this particular script. I hope this can get resolved in the future releases.

[infy-vijayaraghavanl]

I have run into the same problem. When can we expect a fix for this problem?

[wyunchi-ms]

Hi @dansos @infy-vijayaraghavanl , could you please provide more detail about your scenario? Because Set-AzContext will set the TokenCache which is a singleton instance in a azure session. And Foreach-Object -parallel is base on thread model. So it's expected that $context leaks. If you just want to get content, you can use Select-AzContext.

[thgossler]

I don't understand how Select-AzContext could be used for that. Can someone please post example code how to go iterate over 100 subscriptions in parallel and get the number of resource groups in each?

Something like:

$allSubscriptions | ForEach-Object -Parallel { Write-Output "Processing subscription '"$.Name"'" Select-AzContext ... # select the subscription $ only for the current parallel execution scope Write-Output "Subscription" $_.Name "has" (Get-AzResourceGroup).Count "resource groups" }

[Ayanmullick]

I'm trying to get the AzVM deletion activity logs from 100 subscriptions in Parallel. I'm facing the same problem. Select-AzSubscription seems to be switching subscriptions in a non-thread-safe manner causing the below code to be non-idempotent.

$c= Get-AzSubscription|% -Parallel {Select-AzSubscription -SubscriptionId $PSItem.Id|Out-Null; Get-AzLog -StartTime (Get-Date).AddDays(-90)|? {$_.OperationName.Value -EQ 'Microsoft.Compute/virtualMachines/delete'} }

Click to see repro


This shows a different count and output each time. $c.count; $c|select eventTimestamp,EventDataId,resourceGroupName,{$_.operationName.value}

And I get the same issue when using Set-AzContext too. $d= Get-AzSubscription -StartTime (Get-Date).AddDays(-90)|% -Parallel {Set-AzContext -SubscriptionId $PSItem.Id|Out-Null; Get-AzLog|? {$_.OperationName.Value -EQ 'Microsoft.Compute/virtualMachines/delete'} }

[Ayanmullick]

One is able to work around this bug by setting the Azure context inline for the Azure cmdlet. Example below

(Get-AzSubscription).id|ForEach-Object -UseNewRunspace  -Parallel{
    (Get-AzActivityLog -StartTime (Get-Date).AddDays(-90) -ResourceProvider Microsoft.Compute -DefaultProfile  (Set-AzContext -SubscriptionId $PSItem)).Where{$_.OperationName.Value -EQ 'Microsoft.Compute/virtualMachines/delete'}|
        Select-Object eventTimestamp,resourceGroupName,  @{n='AzVMname'; e ={$RId=$_.ResourceId;($RId -replace '[\s\S]*.*(virtualMachines/)')}}
                                                              }|Format-Table -AutoSize  

Thanks @johnthebrit for looking into it.

[srbrills]

Thanks! I did not realize one could utilize the DefaultProfile parameter to provide context without affecting other runspaces, this is very helpful for parallel operations.

[srbrills]

One other note, when attempting to automate resolution of what DefaultProfile values should be passed through, I started working with Get-AzContext a lot more and realized that sometimes a limited number of context objects gets returned at authentication time, and gets cached at that moment. This was especially true when running parallel automation in Azure DevOps agents in pipelines.

To save anyone from a bunch of digging, if you see only ~25 contexts returned when running Get-AzContext -ListAvailable, then you can change this by running Disconnect-AzAccount (to clear the cache), then Connect-AzAccount -MaxContextPopulation <number>. I wrote a function that automatically re-connects service principals in the pipeline and changes the maximum context population, so that functions within a ForEach-Object -Parallel block can dynamically resolve their own contexts from Get-AzContext -ListAvailable and pass this into DefaultProfile parameters.

Reference: https://github.com/Azure/azure-powershell/issues/14482

It would be particularly helpful if the Azure DevOps AzurePowerShell task allowed MaxContextPopulation as a configurable input, but right now it keeps the default 25:

https://github.com/microsoft/azure-pipelines-tasks/blob/214b3bd378c1b5985191e3952da4cc785ee8cd86/Tasks/AzurePowerShellV5/InitializeAz.ps1#L80