Azure / Azure-Functions

1.1k stars 189 forks source link

context leak in parallel threads #2495

Open jaypee1990 opened 3 weeks ago

jaypee1990 commented 3 weeks ago

We have a Azure function to start and stop VMs based on info about VM, its RG and Subscription it reads from Azure storage table when it invocates, and then sets the context to the respective subscription to take action on the VM, however it leaks the context and stays with the context from previous VM and now it cant find that VM in that subscription, which leads to issues.

$schedulesStart | ForEach-Object -Parallel {
        $VMName = $_.Vmname
        $RG = $_.RG
        $Subscription = $_.SubscriptionId
        $VmStatus = ''
        Update-AzConfig -DisplayBreakingChangeWarning $false | Out-Null

        Write-Host "$using:dayType | Start | $Subscription\$RG\$VMName : Time to start the VM"
        Write-Verbose "$using:dayType | Start | $Subscription\$RG\$VMName : Az.Account module version $((get-module Az.Accounts).version.ToString())" -Verbose

        #run action
        try {
            $Retry = 1
            while ($VmStatus.Status -ne "Succeeded") {
                if ($Retry -gt $using:Retries) {
                    throw $ErrorMessage
                }
                Clear-AzContext -Force   # Clear the context before each operation
                $AzContext = Select-AzContext -SubscriptionId $Subscription -ErrorAction SilentlyContinue
                if (!$AzContext) {
                        $ErrorMessage = (Resolve-AzError -Last).Message
                        Write-Warning "$using:dayType | Start | $Subscription\$RG\$VMName : Attempt: $Retry, ErrorMessage: $($ErrorMessage)"
                        start-sleep -Seconds 10 
                    } 
                else {
                    # Check if VM exists
                    $VmExists = Get-AzVM -Name $VMName -ResourceGroupName $RG -ErrorAction SilentlyContinue
                    if (-not $VmExists) {
                    # Skip VM if doesnt exist
                    Write-Warning "VM $VMName in resource group $RG does not exist. Skipping $VMName ."
                    return
                    }
                    $VmStatus = Start-AzVM -name $VMName -ResourceGroupName $RG -ErrorAction SilentlyContinue
                    if (!$VmStatus) {
                        $ErrorMessage = (Resolve-AzError -Last).Message
                        Write-Warning "$using:dayType | Start | $Subscription\$RG\$VMName : Attempt: $Retry, ErrorMessage: $($ErrorMessage)"
                        start-sleep -Seconds 10
                    }
                    else {
                        Write-Host "$using:dayType | Start | $Subscription\$RG\$VMName : Attempt: $Retry, Status: $($VmStatus.Status)"
                    }
                }
                $Retry++
            }
            Add-AzTableRow -table $($using:logTable) -partitionKey $($using:partitionKey) -rowKey ([guid]::NewGuid().tostring()) -property @{"Vmname" = $VMName; "RG" = $RG; "SubscriptionId" = $Subscription; "Action" = "Start"; "Status" = "OK"; "Time" = $($using:currentUTCtime) } | Out-Null
        }
        catch {
            $ErrorName = $_.Exception.GetType().FullName
            $ErrorMessage = $_.Exception.Message
            Add-AzTableRow -table $($using:logTable) -partitionKey $($using:partitionKey) -rowKey ([guid]::NewGuid().tostring()) -property @{"Vmname" = $VMName; "RG" = $RG; "SubscriptionId" = $Subscription; "Action" = "Start"; "Status" = "Failed - $ErrorMessage"; "Time" = $($using:currentUTCtime) } | Out-Null
        }
    } -ThrottleLimit 15

Seems to not work well with parallel threads.

bhagyshricompany commented 1 week ago

Hi @jaypee1990 Thanks for informing please share the logs error.Thanks