Being rate limited on a public repository with codecov token

eXpl0it3r commented 7 months ago

We've been having a lot of issues with rate limiting either with codecov directly or when codecov calls GitHub.

With the v4 GitHub Action, we were hoping that the rate limiting on GitHub's side would go away, but we're now back to being directly rate limited by codecov.

Unfortunately, I can't find any information on what the limits are, so we could actually try to adjust our setup.
Again, this is a public, open source repository, as such the 250 reports per rolling 30 days should by my understanding not apply here.

Can you be more specific of what the rate limits are and is there a way to increase that limit?

Environment

Chrome (not relevant)
GitHub Actions (Linux, Windows, macOS)`
We have around 25-50 reports per PR

To Reproduce

Have some reports already been uploaded from the SFML repository
Create a new pull request in the SFML repository
Wait for the GitHub Actions to run through

Actual behavior The coverage report uploads fail with the v4 action with the message:

debug - 2024-04-10 06:27:11,134 -- Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 429', params={}, description='{\"detail\":\"Request was throttled.\"}'), warnings=[], status_code=429, text='{\"detail\":\"Request was throttled.\"}')"}
error - 2024-04-10 06:27:11,134 -- Upload failed: {"detail":"Request was throttled."}

Expected behavior All the coverage reports are uploaded without issues

Screenshots

Additional context Example GitHub Action runs: https://github.com/SFML/SFML/actions/runs/8623069393/job/23645905253?pr=2932

rohan-at-sentry commented 7 months ago

@eXpl0it3r you're right, the 250 Codecov uploads per rolling 30 shouldn't apply for a public repo. I've flagged this for review with our support team

rohan-at-sentry commented 7 months ago

@eXpl0it3r - how many uploads are you making on the same commit?

eXpl0it3r commented 7 months ago

We have around 25 debug builds across different platforms which generate reports and GitHub triggers builds for the PR and push, so around 50 reports per commit/push.

rohan-at-sentry commented 7 months ago

ok got it... and are you seeing this

debug - 2024-04-10 06:27:11,134 -- Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 429', params={}, description='{\"detail\":\"Request was throttled.\"}'), warnings=[], status_code=429, text='{\"detail\":\"Request was throttled.\"}')"}
error - 2024-04-10 06:27:11,134 -- Upload failed: {"detail":"Request was throttled."}

specifically since the v4 update? iow, Is the 50 reports / commit something that you were always doing (before v4) and you're only now seeing the error?

eXpl0it3r commented 7 months ago

We've had issues with failures before, we usually assumed it was an issue with Codecov being rate limited by GitHub.

As a workaround we just added a retry and tried like 10 times over a period of time. This worked okay-ish, but still caused failures regularly, having us manually retry the failed jobs.

I appreciate the help and hope that highlighting this issue might give you a better understanding of the needs in open source projects.

As we've been fighting Codecov for a while now (see below), we've decided with the latest failures and uncertainties in resolution to move away from Codecov and use Coveralls instead.

amotl commented 7 months ago

Hi. This is right on the spot. We are running into this on CrateDB Toolkit ^1 now the second time in a row, each starting around day 20 of the month (March, and April). We've confirmed with Codecov support that it is apparently a high-traffic repository, receiving many commits.

From my personal judgement, it is not many commits, but sure enough quite a few CI runs, also happening roughly at the same time (we should spread over time), when invoked per nightly CI runs. In this spirit, we are sure this specific repository is quite heavy in terms of aggregating load over the course of a month, so we will consider jumping on a paid plan in order to support Codecov.

Saying this, thanks a stack for supporting all the other open source projects out there. We hope you can keep it like this. [^2]

[^2]: ... at least, for all of them having common/average usage patterns, but also for special operations who just can't afford it.

stronk7 commented 5 months ago

Hi,

just to add a little more about this issue and my experience... few days ago (using the free / open source access), I started to do some experiments with our main repository @ github (Moodle), with my own fork (stronk7/moodle) and the Flags feature.

To do so I just split our mammoth PHPUnit runs into the 429 test suites that we have, generating a clover coverage for each and went uploading them, each one with its own flag defined. I'm using the codecov CLI locally, with commands like this:

codecov --verbose upload-process --fail-on-error \
          -t "${CODECOV_TOKEN}" \
          -r "stronk7/moodle" \
          --git-service=github \
          -f "build/${FLAG}.xml" \
          --disable-search \
          -F "${FLAG}" \
          -C "${COMMIT}" \
          -n "${FLAG}-${SHORTCOMMIT}"

And everything went ok until I started to get those "Request was throttled" responses:

Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 429', params={}, description='{\"detail\":\"Request was throttled.\"}'), warnings=[], status_code=429, text='{\"detail\":\"Request was throttled.\"}')"}

Then started to search for the error and, wow, was really hard to find any info. Few minutes ago I discovered this issue.

I've tried again today and I consistently continue getting the 429 error above. And, for sure, the repo is public, open source and all those things.

Little extra detail, few minutes ago I've updated my (personal) account to the "Teams" (2 seats minimum, no matter I'm a solo developer in this testing case, heh) to see if that made some difference... and, bad luck, it still is returning the very same error.

So, really, it would be great if we can get some extra information or confirmation about this issue. Is it intended behaviour or is it a bug? Why switching to a paid plan doesn't make any difference? As said, I've not been able to find any place where it's stated that the 250 (or < 429, I honestly haven't counted how many of our split reports were uploaded) is a limit for public / open source repos.

I personally like what Codecov provides and how it integrates with GH workflows and others (we are already using it for a growing number of Moodle-related open source repos), but the throttling that we are experiencing with my 429-uploads split is a little bit worrying... specially if that's some sort of global counter that affects to other repos belonging to the same account.

Thanks in advance for any clarifications and keep the cov-ball going!

Ciao :-)

PS: Believe me, I'm not joking, both the error code and the number of test suites we have is 429, a real coincidence! 😆

$ grep -oP '(?<=<testsuite name=")[^"]+' phpunit.xml | wc -l
429

codecov / feedback

Being rate limited on a public repository with codecov token #322