Back-off policy (for 429 Errors) throws an exception because the response from the call to Invoke-WebRequest() is not in the expected format.

[MrDRamos]

The back-off policy (for 429 Error) throws an exception because the response from the call to Invoke-WebRequest() is not in the expected format. I was running with cosmosDB version 4.5.0. I verified the code in the same in version: 4.6.0-preview0003

This is on Azure VM running Windows Server 2012R2 running PowerShell version 5.1.14409.1005

$PSVersionTable Name Value

---

PSVersion 5.1.14409.1005 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.14409.1005 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1

I get the following error: "Cannot index into a null array." Stacktrace: at Invoke-CosmosDbRequest, C:\Program Files\WindowsPowerShell\Modules\cosmosdb\4.5.0\CosmosDB.psm1: line 844 at New-CosmosDbDocument, C:\Program Files\WindowsPowerShell\Modules\cosmosdb\4.5.0\CosmosDB.psm1: line 4360

I fixed the code to avoid de-referencing the $null value as follows: <# The exception was caused by exceeding provisioned throughput so determine is we should delay and try again or exit

>

[System.Int32] $retryAfter = 0 $RetryInfo = $_.Exception.Response.Headers | Where-Object -Property Key -EQ 'x-ms-retry-after-ms' if ($RetryInfo) { $retryAfter = $RetryInfo.Value[0] }

[System.Int32] $retryAfter = ($_.Exception.Response.Headers | Where-Object -Property Key -eq 'x-ms-retry-after-ms').Value[0]

[PlagueHO]

Thanks for raising this @MrDRamos. Looks like an easy fix and thank you for. What is strange though is why the x-ms-retry-after-ms header isn't being passed with a 429 - do you see this header in the response message for the 429?

[MrDRamos]

Just did a little more testing. The error happens when we write a large document (1Mb+ size) to cosmosDb. Typically the large doc is written but subsequent write operations that occur immediately afterword’s fail with a 429 error. I set a breakpoint in the catch statements here are the details:

D:\Builds\olcportal\Modules\wip> $_.Exception.Response

IsMutuallyAuthenticated : False Cookies : {} Headers : {x-ms-gatewayversion, x-ms-activity-id, x-ms-retry-after-ms, x-ms-schemaversion...} SupportsHeaders : True ContentLength : 193 ContentEncoding : ContentType : application/json CharacterSet : Server : Compute LastModified : 7/5/2022 11:31:07 AM StatusCode : 429 StatusDescription : Too Many Requests ProtocolVersion : 1.1 ResponseUri : https://xxxxx.documents.azure.com/dbs/devtest/colls/SpeedTest/docs Method : POST IsFromCache : False

D:\Builds\olcportal\Modules\wip> $_.Exception.Response.Headers | fl *

x-ms-gatewayversion x-ms-activity-id x-ms-retry-after-ms x-ms-schemaversion lsn x-ms-request-charge x-ms-quorum-acked-lsn x-ms-substatus x-ms-current-write-quorum x-ms-current-replica-set-size x-ms-xp-role x-ms-global-Committed-lsn x-ms-number-of-read-regions x-ms-transport-request-id x-ms-cosmos-llsn x-ms-cosmos-quorum-acked-llsn x-ms-serviceversion Content-Length Content-Type Date Server

D:\Builds\olcportal\Modules\wip> ($_.Exception.Response.Headers | Where-Object -Property Key -EQ 'x-ms-retry-after-ms') -eq $null True

In the workaround I sent, the delay time set in the BackOffPolicy is used since the x-ms-retry-after-ms contet is $null

From: Daniel @.> Sent: Friday, July 1, 2022 6:22 PM To: @.> Cc: David @.>; @.> Subject: Re: [PlagueHO/CosmosDB] Back-off policy (for 429 Errors) throws an exception because the response from the call to Invoke-WebRequest() is not in the expected format. (Issue #458)

Thanks for raising this @MrDRamoshttps://github.com/MrDRamos. Looks like an easy fix and thank you for. What is strange though is why the x-ms-retry-after-ms header isn't being passed with a 429 - do you see this header in the response message for the 429?

— Reply to this email directly, view it on GitHubhttps://github.com/PlagueHO/CosmosDB/issues/458#issuecomment-1172759313, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJADKIYQTYVY7FYL5WQERFDVR5VQZANCNFSM52N3A4VA. You are receiving this because you were mentioned.Message ID: @.***>

[PlagueHO]

Thanks @MrDRamos for the extra info. I'll try and come up with a unit test and apply the fix this weekend. Been a bit snowed under this week.