Open phill-holbrook opened 9 months ago
I think I've found a solution for this by removing Invoke-WebRequest
and using System.Net.Http.HttpClient
directly.
function Get-APIResource {
param(
[Parameter (Mandatory=$true)]$Resource,
[Parameter (Mandatory=$true)]$SearchQuery
)
$BaseURL = ""
$httpClient = New-Object System.Net.Http.HttpClient
$httpClient.DefaultRequestHeaders.Add("ApiCode", $env:ApiCode)
$httpClient.DefaultRequestHeaders.Add("UserName", $env:APIUser)
$httpClient.DefaultRequestHeaders.Add("APIKey", $env:APIKey)
$ResourceURL = $BaseURL + $Resource + '/query?search=' + $SearchQuery
$i = 1
try {
do {
Write-Warning "Getting page $i of $Resource"
$responseStream = $httpClient.GetStreamAsync($ResourceURL).Result
$streamReader = [System.IO.StreamReader]::new($responseStream)
$responseContent = $streamReader.ReadToEnd()
$Response = $ResponseContent | ConvertFrom-Json
If ($Response.items){
Write-Warning "($i) Retrieved $($Response.items.count) items"
$Response.items | ForEach-Object{
$_
#$ReturnList.Add($_)
}
}
If ($Response.item){
Write-Warning "($i) Retrieved $($Response.item.count) ITEM"
$Response.item | ForEach-Object{
$_
#$ReturnList.Add($_)
}
}
$ResourceURL = $Response.pagedetails.nextPageUrl
If ($null -eq $ResourceURL){
Write-Warning "($i) ResourceURL is null"
} Else {
Write-Warning "($i) Next page URL: $ResourceURL"
}
$i++
} while ($null -ne $ResourceURL)
}
catch {
Write-Error "Connecting to API Failed."
throw "API Error: $($_.Exception.Message)"
}
}
I have no idea why I'm unable to reproduce the error calling the class directly. My guess is it's some perfect storm between the language runtime, Invoke-WebRequest
, and the remote API in question. Hope this helps someone else.
I have a function that uses Invoke-WebRequest in order to pull data from an external API. Here's a sanitzed version of the function:
I'm using it in an Activity function, like so:
And this is being called by an Orchestrator function:
When I look at Log Stream in the Azure Function, I can see the progress of
Invoke-WebRequest
. When it runs normally, it looks like this:However, when it has a problem, it will halt after a random number of bytes read. Verbose logs will continue to output from normal operation of the Azure Function. The Activity Function times out after 10 minutes, at which point the Orchestrator finishes.
Normally, the API request takes under 30 seconds to complete -- usually it completes in under 10 seconds. But since the web stream from
Invoke-WebRequest
is hanging the Activity Function up, I cannot implement Durable Retry or Durable Wait in the Orchestrator function. I've tried wrapping eachInvoke-WebRequest
inStart-ThreadJob
withinGet-APIResource
and usingWait-Job
with a 30 second timeout. I'm able to properly throw an error from the API function to the Activity function, and the Orchestrator function catches it, but the Activity Function still sits there and waits until 10 minutes to time out because the thread job is locked up thanks to theInvoke-WebRequest
stream.I cannot reproduce this problem locally (via manual execution of
Get-APIResource
or via Azure Function debugging in VS Code) or on an Azure VM. I can fairly reliably reproduce this problem within Azure Automation Accounts / Runbooks and Azure Functions.I've been chasing this error down since at least September 2023. I'm happy to share any other required information to help track this down.