SCRT-HQ / PSGSuite

Powershell module for Google / G Suite API calls wrapped in handy functions. Authentication is established using a service account via P12 key to negate the consent popup and allow for greater handsoff automation capabilities
https://psgsuite.io/
Apache License 2.0
234 stars 68 forks source link

Implement retries with exponential backoff for long running commands like Get-GSDriveFileList #390

Open Celestica-Edward opened 2 months ago

Celestica-Edward commented 2 months ago

Describe the bug Commands such as Get-GSDriveFileList calls Google APIs numerous times if the user has many files. Any of these API calls can transiently fail with 500 internal server error (http://disq.us/t/3ib2tne) or any other error. Retrying from the parent script would mean the file list needs to be from the beginning again, wasting lots of time.

To Reproduce Steps to reproduce the behavior:

  1. Call Get-GSDriveFileList against a user with many thousands of files and if you're unlucky, an error is encountered. Capture Expected behavior The loop at https://github.com/SCRT-HQ/PSGSuite/blob/main/PSGSuite/Public/Drive/Get-GSDriveFileList.ps1#L170 can be modified to include retries, especially for 5xx errors. Other error codes such as 4xx should not be retried.

A possible example can be:

                $MaxRetries = 5
                $RetryDelaySeconds = 2
                $retryCount = 0
                while ($retryCount -lt $MaxRetries) {
                    try {
                        $result = $request.Execute()
                        break
                    }
                    catch {
                        Write-Verbose "Failed, retrying..."
                        Write-Verbose $_
                        $retryCount++
                        Write-Verbose $("Sleeping " + ([math]::Pow($RetryDelaySeconds, $retryCount)) + " seconds")
                        Start-Sleep -Seconds ([math]::Pow($RetryDelaySeconds, $retryCount))
                    }
                }
                if ($retryCount -eq $MaxRetries) {
                    Write-Verbose "Failed after $retryCount attempts"
                    throw $_
                }

Capture2