[Bug]: Ratelimiting?

[sultanberisa]

What happened?

I am building an automated packaging workflow using evergreen. At some point, i noticed that my variable which would hold the download url which is dynamically retrieved via evergreen, was null. Upon checking closer i can see that i get ratelimited:

PS C:\NIP_Software\Scripts> $DownloadURL = Get-EvergreenApp -Name $AppName | Where-Object { if (!$_.Architecture -or $_.Architecture -eq "x64") {$true} else {$false} } | Where-Object { if (!$_.Channel -or $_.Channel -eq $sourcechannel) {$true} else {$false} } | Where-Object { if (!$_.Type -or $_.Type -eq $sourcepackagetype) {$true} else {$false} } | Where-Object { if (!$_.Platform -or $_.Platform -eq $sourceplatform) {$true} else {$false} } | Where-Object { if (!$_.Language -or $_.Language -eq $sourcelanguage) {$true} else {$false} } | Select -ExpandProperty URI | Select-Object -First 1
WARNING: Get-GitHubRateLimit: Requests to GitHub are being rate limited.
PS C:\NIP_Software\Scripts> Get-EvergreenApp -Name $appname
WARNING: Get-GitHubRateLimit: Requests to GitHub are being rate limited.

Version     URI
-------     ---
RateLimited https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting

PS C:\NIP_Software\Scripts> $AppName = GitForWindows
GitForWindows : The term 'GitForWindows' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:12
+ $AppName = GitForWindows
+            ~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (GitForWindows:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

PS C:\NIP_Software\Scripts> $Arch = "x64"
PS C:\NIP_Software\Scripts> $AppType = "EXE"
PS C:\NIP_Software\Scripts>
PS C:\NIP_Software\Scripts> $ThisApp = (Get-EvergreenApp $($AppName) | Where-Object {$_.Architecture -eq $($Arch) -and $_.Type -eq $($AppType).ToLower() -and $_.URI -notlike "*PortableGit*"})
WARNING: Get-GitHubRateLimit: Requests to GitHub are being rate limited.
PS C:\NIP_Software\Scripts> #[version]$LatestVersion = $ThisApp.Version
PS C:\NIP_Software\Scripts> $SourceUrl = $ThisApp.URI
PS C:\NIP_Software\Scripts> # Evergreen version returns 3 digit version so obtain 4 digit version from URI
PS C:\NIP_Software\Scripts> [version]$LatestVersion = (((($ThisApp).URI -split "/")[7]) -replace '^v') -replace '.windows'
Cannot convert value "" to type "System.Version". Error: "Versjonsstrengdelen var for kort eller for lang."
At line:1 char:1
+ [version]$LatestVersion = (((($ThisApp).URI -split "/")[7]) -replace  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : MetadataError: (:) [], ArgumentTransformationMetadataException
    + FullyQualifiedErrorId : RuntimeException

How can i avoid this / solve this somehow? If i get ratelimited during a workflow it would be pretty stupid and would require manual action, thus removing the whole point of the "workflow" haha

Version

2309.840

What PowerShell edition/s are you running Evergreen on?

Windows PowerShell

Which operating system/s are you running Evergreen on?

Windows 10+

Have you reviewed the documentation?

• [X] Troubleshooting at: https://stealthpuppy.com/evergreen/troubleshoot/
• [X] Known issues at: https://stealthpuppy.com/evergreen/issues/

Verbose output

PS C:\NIP_Software\Scripts> $DownloadURL = Get-EvergreenApp -Name $AppName | Where-Object { if (!$_.Architecture -or $_.Architecture -eq "x64") {$true} else {$false} } | Where-Object { if (!$_.Channel -or $_.Channel -eq $sourcechannel) {$true} else {$false} } | Where-Object { if (!$_.Type -or $_.Type -eq $sourcepackagetype) {$true} else {$false} } | Where-Object { if (!$_.Platform -or $_.Platform -eq $sourceplatform) {$true} else {$false} } | Where-Object { if (!$_.Language -or $_.Language -eq $sourcelanguage) {$true} else {$false} } | Select -ExpandProperty URI | Select-Object -First 1^C
PS C:\NIP_Software\Scripts> $DownloadURL = Get-EvergreenApp -Name $AppName -Verbose | Where-Object { if (!$_.Architecture -or $_.Architecture -eq "x64") {$true} else {$false} } | Where-Object { if (!$_.Channel -or $_.Channel -eq $sourcechannel) {$true} else {$false} } | Where-Object { if (!$_.Type -or $_.Type -eq $sourcepackagetype) {$true} else {$false} } | Where-Object { if (!$_.Platform -or $_.Platform -eq $sourceplatform) {$true} else {$false} } | Where-Object { if (!$_.Language -or $_.Language -eq $sourcelanguage) {$true} else {$false} } | Select -ExpandProperty URI | Select-Object -First 1
VERBOSE: Function path: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2309.840\Apps\Get-NotepadPlusPlus.ps1
VERBOSE: Function exists: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2309.840\Apps\Get-NotepadPlusPlus.ps1.
VERBOSE: Dot sourcing: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2309.840\Apps\Get-NotepadPlusPlus.ps1.
VERBOSE: Get-FunctionResource: read application resource strings from [C:\Program
Files\WindowsPowerShell\Modules\Evergreen\2309.840\Manifests\NotepadPlusPlus.json]
VERBOSE: Calling: Get-NotepadPlusPlus.
VERBOSE: Get-GitHubRateLimit: Checking for how many requests to the GitHub API we have left.
VERBOSE: GET https://api.github.com/rate_limit with 0-byte payload
VERBOSE: received 465-byte response of content type application/json; charset=utf-8
VERBOSE: Get-GitHubRateLimit: We have 0 requests left to the GitHub API in this window.
VERBOSE: Get-GitHubRateLimit: Rate limit window resets at: 11.09.2023 20:08.
WARNING: Get-GitHubRateLimit: Requests to GitHub are being rate limited.
PS C:\NIP_Software\Scripts>

[aaronparker]

You can authenticate to the GitHub API, and save the authentication token to $Env:GH_TOKEN or $Env:GITHUB_TOKEN. Evergreen will then use the token when querying the API, and you won't be rate limited.

See https://docs.github.com/en/rest/overview/authenticating-to-the-rest-api

[sultanberisa]

You can authenticate to the GitHub API, and save the authentication token to $Env:GH_TOKEN or $Env:GITHUB_TOKEN. Evergreen will then use the token when querying the API, and you won't be rate limited.

See https://docs.github.com/en/rest/overview/authenticating-to-the-rest-api

Saving my Fine-grained personal access token into $Env:GH_TOKEN or $Env:GITHUB_TOKEN in PowerShell is not working. Still recieving rate limits

[g3rhard]

Same for me, got this today.

Not sure, but this part probably could be improved:

                # If GITHUB_TOKEN or GH_TOKEN exists, let's add that to the API request
                if (Test-Path -Path "env:GITHUB_TOKEN") {
                    $params.Headers = @{ Authorization = "token $env:GITHUB_TOKEN" }
                }
                elseif (Test-Path -Path "env:GH_TOKEN") {
                    $params.Headers = @{ Authorization = "token $env:GH_TOKEN" }
                }

to something like this:

        if ($env:GITHUB_TOKEN) {
            $params.Headers = @{ Authorization = "token $env:GITHUB_TOKEN" }
        }
        elseif ($env:GH_TOKEN) {
            $params.Headers = @{ Authorization = "token $env:GH_TOKEN" }
        }

In my case current version didn't work as expected:

$env:GH_TOKEN="github_pat_1XXXX"

> Get-EvergreenApp -Name "AWSCLI" -Verbose
VERBOSE: Function path: C:\Users\admin\Documents\PowerShell\Modules\Evergreen\2407.1230\Apps\Get-AWSCLI.ps1
VERBOSE: Function exists: C:\Users\admin\Documents\PowerShell\Modules\Evergreen\2407.1230\Apps\Get-AWSCLI.ps1.
VERBOSE: Dot sourcing: C:\Users\admin\Documents\PowerShell\Modules\Evergreen\2407.1230\Apps\Get-AWSCLI.ps1.
VERBOSE: Get-FunctionResource: read application resource strings from [C:\Users\admin\Documents\PowerShell\Modules\Evergreen\2407.1230\Manifests\AWSCLI.json]
VERBOSE: Calling: Get-AWSCLI.
VERBOSE: Invoke-EvergreenWebRequest: Adding ContentType.
VERBOSE: Invoke-EvergreenWebRequest: Invoke-WebRequest parameter: UseBasicParsing: True.
VERBOSE: Invoke-EvergreenWebRequest: Invoke-WebRequest parameter: Uri: https://api.github.com/repos/aws/aws-cli/tags.
VERBOSE: Invoke-EvergreenWebRequest: Invoke-WebRequest parameter: ContentType: application/json; charset=utf-8.
VERBOSE: Invoke-EvergreenWebRequest: Invoke-WebRequest parameter: UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.2592.87.
VERBOSE: Invoke-EvergreenWebRequest: Invoke-WebRequest parameter: Method: Default.
VERBOSE: Requested HTTP/1.1 GET with 0-byte payload
VERBOSE: Received HTTP/1.1 281-byte response of content type application/json
WARNING: Invoke-EvergreenWebRequest: Response status code does not indicate success: 403 (rate limit exceeded)., with: https://api.github.com/repos/aws/aws-cli/tags.
WARNING: Invoke-EvergreenWebRequest: For troubleshooting steps see: https://stealthpuppy.com/evergreen/troubleshoot/.
WARNING: Run 'Get-EvergreenApp -Name "AWSCLI" -Verbose' to review additional details for troubleshooting.
Invoke-WebRequest: C:\Users\admin\Documents\PowerShell\Modules\Evergreen\2407.1230\Private\Invoke-EvergreenWebRequest.ps1:123
Line |
 123 |              $Response = Invoke-WebRequest @params
     |                          ~~~~~~~~~~~~~~~~~~~~~~~~~
     |  {   "message": "API rate limit exceeded for 147.161.248.199. (But here\u0027s the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",   "documentation_url":
     | "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting" }

[aaronparker]

AWSCLI isn't using the internal Get-GitHubRepoRelease function which supports tokens. I'll have to update that app