The Steampipe GitHub plugin should already be doing its own request throttling, but we're clearly still bumping into GitHub's limits sometimes:
ERROR: rpc error: code = Unknown desc = github: GET https://api.github.com/repos/nextstrain/react-sidebar/actions/workflows?per_page=100: 403 API rate limit of 1000 still exceeded until 2024-02-15 04:58:05 +0000 UTC, not making remote request. [rate reset in 7m26s]
It does really seem like this should work out of the box. Worth checking source that it should. And if so, perhaps it's not because it can't properly track rates when we're starting from an empty Steampipe state every time (new Steampipe Pg service, new install, new workflow job, etc.) and doing so frequently?
Since "every 5 minutes" GitHub Actions workflows are not reliably actually every 5 minutes, we could opt instead to launch a long-running workflow on GitHub Actions every ~6 hours or something (with concurrency to have a shallow queue) and do the high frequency polling within it. This would help solve thought 1 too, if indeed that's an issue.
The keen-eyed may note that the above example error references workflows for the nextstrain/react-sidebar repo. They may wonder why we're looking at that repo at all. It's a good question! The answer is that doing so is a) too easy and b) means we don't have to update a query/source code here when a new pathogen repo starts using pathogen-repo-build. But we could probably rewrite bits of the query to avoid making unnecessary requests. I'd count this as a measure of last resort, though.
The Steampipe GitHub plugin should already be doing its own request throttling, but we're clearly still bumping into GitHub's limits sometimes:
The Steampipe feature to dig into is "limiters": https://steampipe.io/docs/guides/limiter
Maybe also read a bit about caching: https://steampipe.io/docs/guides/caching (but really, I don't expect the cache to help us here: we want the fresh data)
A few thoughts:
It does really seem like this should work out of the box. Worth checking source that it should. And if so, perhaps it's not because it can't properly track rates when we're starting from an empty Steampipe state every time (new Steampipe Pg service, new install, new workflow job, etc.) and doing so frequently?
Since "every 5 minutes" GitHub Actions workflows are not reliably actually every 5 minutes, we could opt instead to launch a long-running workflow on GitHub Actions every ~6 hours or something (with concurrency to have a shallow queue) and do the high frequency polling within it. This would help solve thought 1 too, if indeed that's an issue.
The keen-eyed may note that the above example error references workflows for the nextstrain/react-sidebar repo. They may wonder why we're looking at that repo at all. It's a good question! The answer is that doing so is a) too easy and b) means we don't have to update a query/source code here when a new pathogen repo starts using
pathogen-repo-build. But we could probably rewrite bits of the query to avoid making unnecessary requests. I'd count this as a measure of last resort, though.