GitHub Advisory Mirroring recursion does not properly handle secondary rate limits

[officerNordberg]

Current Behavior

GitHub Mirroring fails nearly daily and is not resilient. While troubleshooting after weeks of ~30 failures/per day due I believe to the recursive nature of the mirroring I finally added a sleep statement between all graphql calls which resulted getting the first updates to the VULNERABILITY table since the 11th, the last day it ran without incident. It's hard to confirm this as the CREATED column is not populated on the table but UPDATED is and inferring Descending Primary Keys as "created dates" there are large gaps in GHSA primary keys that correlate to the days I received notifications of failed mirror attempts. I think the recursion is trying to handle this but it just results in 30 or more error Notifications being sent instead of just 1.

I added some additional details to the error message.

2024-10-21 16:49:15,024 ERROR [GitHubAdvisoryMirrorTask] An error was encountered retrieving advisories with HTTP Status : 403 Forbidden 
{  "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits",  
"message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. 
If you reach out to GitHub Support for help, please include the request ID DEAD:DEED39:7ADE8B:ED56A1:67168609."}

curling my user I see these headers

< x-github-media-type: github.v4; format=json
< x-ratelimit-limit: 5000
< x-ratelimit-remaining: 4845
< x-ratelimit-reset: 1729532625
< x-ratelimit-used: 155
< x-ratelimit-resource: graphql

Attempted to get feedback on this issue as a discussion first. https://github.com/DependencyTrack/dependency-track/discussions/4239

Steps to Reproduce

1. Enable GitHub Mirroring and notifications when it fails.

Expected Behavior

Secondary rate limits should be handled with a retry with backoff strategy.

Dependency-Track Version

4.11.x

Dependency-Track Distribution

Container Image

Database Server

PostgreSQL

Database Server Version

11.22

Browser

Google Chrome

Checklist

• [x] I have read and understand the contributing guidelines
• [x] I have checked the existing issues for whether this defect was already reported

[nscuro]

Been meaning to switch to https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/open-vulnerability-clients#github-security-advisory-example for this anyway. We already use that lib for the NVD REST API integration and it handles retries, also for rate limiting, behind the scenes. In doing this, we should also switch to incremental mirroring so we're not re-downloading everything everyday, which is what we're currently doing.