Handle rate limiting errors for GitHub API

[Vyom-Yadav]

Describe the issue

Currently, we don't handle rate limit exceeded errors from the GitHub API. From GitHub API Docs:

Continuing to make requests while you are rate limited may result in the banning of your integration.

GitHub suggests Best practices for using the REST API, which mentions handling the rate limit exceeded errors.

Discord Thread: https://discord.com/channels/1184987096302239844/1185287949240242258/1201083912521252885

• [x] Add client caching to handle rate limiting (https://github.com/stacklok/minder/pull/2271)
• [ ] Update all endpoints to handle rate-limiting errors

To Reproduce

No response

What version are you using?

main

[Vyom-Yadav]

Rate-limiting errors are on a token/user basis. Blocking the goroutine by storing some state about the token should prevent Minder from making repeated requests using that token, preventing the integration from getting banned.

Currently, Minder uses gochannel and SQL pub-sub implementation from Watermill to handle incoming events. Both are single channels with head-of-line blocking, which means that blocking in middleware/handler blocks all processing. This means blocking any goroutine (created by event handlers/middleware) using current pub-sub implementations would block the complete event processing.

However, given the current scale, getting rate limited is rare, and over the last month we've observed zero 403 or 429 responses (See this comment on discord), so having blocking logic until we replace with some other pub-sub implementation shouldn't be a problem.

[Vyom-Yadav]

/assign