f0oster
commented
1 year ago This would be a useful feature.
I've implemented this feature in my own modules in the past in a fairly naïve way. I didn't find it necessary to track the request chain at the point of failure (ie, the current data, the nextLink for the next query, etc) and just elected start the request chain again from the top level.
If you were looking for some inspiration, I've thrown together a quick example (untested) which roughly emulates my approach.
../private/Invoke-HTTPGenericApiQuery .ps1, generic method to communicate with the API, which all the public API functions call
function Invoke-HTTPGenericApiQuery {
param(
$Method,
$Resource,
$Body,
$MaxRetries = 5,
$API_URL # base url for API,
)
$PropertySplat = @{
Headers = $Header # usually I'd inject headers via a private module/script scoped variable
Uri = "$API_URL/$Resource"
Method = $Method
ContentType = "application/json"
}
if ($Method -ne "GET" -and $Method -ne "DELETE") {
$PropertySplat.add("Body", $Body)
}
try {
for ($RetryCount = 0; $RetryCount -lt $MaxRetries; $RetryCount++) {
# can likely transplant your existing code here this is just a naive example
$Responses = [System.Collections.Generic.List[PSObject]]::new()
$Responses.Add((Invoke-RestMethod @PropertySplat))
# implement your nextlink handling / custom filtering / data concatenation logic below
# no HTTP exception was thrown, we don't need to retry :)
return $Responses
}
# We met our MaxRetries
throw("TooManyRetriesError") # can implement your own error types if you like
}
catch {
# Ideally handle all other possible non-HTTP exceptions first
# As an example, if the internal function throws an error that the user hasn't connected yet,
# we'll rethrow to the caller
if ($_ -like "Please connect using*") {
throw($_) # rethrow the error, or your own custom error type to be handled by the caller
}
if ($_ -like "TooManyRetries*") {
throw($_) # rethrow the error, or your own custom error type to be handled by the caller
}
# The implementation of Invoke-WebRequest and Invoke-RestMethod in PS 5.1 is absolutely awful
# You will need to consider forking logic here to support multiple platforms
# A quick and dirty approach would be something like below:
if ($PSVersionTable.PSVersion -like "5.*") {
# It would be best to confirm that $_.Exception actually contains a response first :)
$HTTPResponse = $_.Exception.Response.GetResponseStream()
$SR = New-Object System.IO.StreamReader($HTTPResponse)
$SR.BaseStream.Position = 0
$SR.DiscardBufferedData()
$ResponseBody = $SR.ReadToEnd();
if ($_.Exception.Response.StatusCode -eq 404) {
throw($_) # rethrow the error, or your own custom error type to be handled by the caller
}
if ($_.Exception.Response.StatusCode -eq 404) {
throw($_) # rethrow the error, or your own custom error type to be handled by the caller
}
if ($_.Exception.Response.StatusCode -eq 429) {
$RetryAfter = $_.Exception.Response.Headers["Retry-After"]
Write-Error "HTTP Response 429 - Too many requests, retrying in $RetryAfter seconds (Retry-After HTTP header)"
Write-Error "The HTTP response body was:`n $ResponseBody"
Start-Sleep -Seconds $RetryAfter
}
}
else {
# PS 7/non-Windows PS code here to handle HTTP exceptions
}
}
}
insomniacc
A useful enhancement would be to auto handle rate limiting errors, retry x times with the backoff provided in the
X-RateLimit-ResetorRetry-Afterheader, if the request still cannot be sent it should throw an exception.https://docs.servicenow.com/bundle/sandiego-application-development/page/integrate/inbound-rest/concept/inbound-REST-API-rate-limiting.html