search

codecov / feedback

A place to discuss feedback about the pull request and web product experience.
36 stars 7 forks source link

Document how to properly setup Codecov Github Action for OpenSource repositories #301

Open ST-DDT opened 8 months ago

ST-DDT commented 8 months ago

Describe the bug

PRs from external contributors fail CI.

The PR is happening in a forked repo. Using tokenless upload. Tokenless has reached GitHub rate limit.

Environment (please complete the following information):

To Reproduce

Steps to reproduce the behavior:

1) Go to an OpenSource project with CodeCov GitHub Action 2) Open a PR as external contributor 3) The CI will very likely fail (~10 CI failures, sometimes multiple in a row)

External contributors have a near 100% chance at failing. Internal contributors have a near 0% chance at failing.

Expected behavior

The CI is expected to work. Or alternatively the guide describing how to setup the action should provide hints on how to deal with the error.

Screenshots

info - 2024-03-12 15:43:46,354 -- ci service found: github-actions
info - 2024-03-12 15:43:46,690 -- The PR is happening in a forked repo. Using tokenless upload.
info - 2024-03-12 15:43:47,076 -- Process Commit creating complete
error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}
Error: Codecov: Failed to properly create commit: The process '/home/runner/work/_actions/codecov/codecov-action/54bcd8715eee62d40e33596ef5e8f0f48dbbccab/dist/codecov' failed with exit code 1

Additional context

Created on request here:

Affected CI pipelines:

kdarkhan commented 7 months ago

It might be hard to find the workaround I came up with from the linked issue, adding it here if that becomes useful.

I ended up using 2 workflow approach which seems to work pretty stable with token based auth for forked repos.

This workflow uploads coverage report to Github artifacts and this workflow uses that report and the token to upload it to codecov.

adamjstewart commented 7 months ago

Any updates on this? We're seeing a 100% failure rate on most recent PRs. Is the open source community expected to transition to a different coverage provider, or are there future plans to once again support forks?

Screenshot 2024-04-13 at 19 10 17
rohan-at-sentry commented 7 months ago

@ST-DDT thanks for raising.

For context (which you probably already realized from the screenshot - but may benefit others), v4 allows tokenless uploads for forks ; enabling open-source contributors to upload coverage.

Because this happens without a token, Codecov uploads from forks upload using Codecov's own token - you can see how quickly that can get consumed leading to a failure. That is most likely what is happening when you see 

error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}

In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?

Also - I'm curious if the workaround here worked for you

ST-DDT commented 7 months ago

In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?

I think the rate remained the same. It just happened a lot at the time of reporting (and for an extended period of time). Previously it was an undefimed error, now it is a specific error.

Also - I'm curious if the workaround here worked for you

No, we are currently in a feature freeze and thus have very few external contributions. We generally have a low need for code coverage and thus consider removing it instead due to the additional work required.

adamjstewart commented 7 months ago

I tried to use the above workaround. Didn't try the full thing with hundreds of lines of JavaScript code, but it was too complicated to get working with the usual upload/download artifact actions. TorchGeo gets 99% external contributions, so we're very keen on getting this working.

rohan-at-sentry commented 7 months ago

Hey folks cc @ST-DDT @adamjstewart

We're exploring the best path forward for a fix here. Here's the current thinking - https://github.com/codecov/engineering-team/issues/1574

We'd expect this to be ~ 4 weeks of work.. I'll report back here once this is complete. We're shooting for something that doesn't require more config.

thank you for your patience.

rohan-at-sentry commented 6 months ago

Meanwhile, we've better documented the tokenless flow in our docs . In general, there shouldn't be a need to implement workarounds to pass the token to forks that can then use it to upload to Codecov.

LebedevRI commented 6 months ago

It would be nice to expand

(and the latest version of the Codecov Action)

with an information about the actual first version that supports that behaviour.

adamjstewart commented 6 months ago

Could also replace:

there may be cases when uploading to Codecov via tokenless fails

with:

there may be cases when uploading to Codecov via tokenless works

as the former is much more common (at the moment) 😏

But seriously though, thanks for all of your hard work to improve the situation. I know how difficult it is when GitHub refuses to increase the rate limit for a token and security is involved. I'm sure you're drowning in bug reports at the moment. Looking forward to a working solution, and happy to help test on our repo!

rohan-at-sentry commented 6 months ago

It would be nice to expand

(and the latest version of the Codecov Action)

with an information about the actual first version that supports that behaviour.

Agreed, updated

rohan-at-sentry commented 6 months ago

Could also replace:

there may be cases when uploading to Codecov via tokenless fails

with:

there may be cases when uploading to Codecov via tokenless works

as the former is much more common (at the moment) 😏

image

But seriously though, thanks for all of your hard work to improve the situation. I know how difficult it is when GitHub refuses to increase the rate limit for a token and security is involved. I'm sure you're drowning in bug reports at the moment. Looking forward to a working solution, and happy to help test on our repo!

Thanks for your patience as we work through this. <3

jotak commented 5 months ago

Hi folks, I was wondering if it would be possible to use codecov on PRs (via pull_request GH event) only for posting comment and without uploading the report? Report uploads being done via another workflow, on push events, which doesn't have the same impact on secrets visibility. If that's technically doable, I guess it would be beneficial not only to fix the failures we're currently seeing, but also in terms of lowering the traffic & attempts between codecov and github api ?

adamjstewart commented 5 months ago

That is the workaround discussed in https://github.com/codecov/feedback/issues/126. See https://github.com/codecov/feedback/issues/126#issuecomment-1931862637 for why that's currently way too difficult to get working properly. Of course, if there was a documented solution or simplification of this that worked, that would be fine. But a solution that allowed PR coverage to be uploaded without a secret and upstream repo coverage to be only uploaded with a secret would also be fine with me.

jotak commented 5 months ago

@adamjstewart I was thinking something slightly different - but maybe this is naive as I don't really know how codecov works under the hood :

adamjstewart commented 5 months ago

A comment would be better than nothing, but the ability to view the diff and missing coverage on the codecov website would be better. I don't think it would be a security issue to upload PR coverage without a secret, as long as a secret is still required for protected branches like main.

kdarkhan commented 5 months ago

Based on previous comments it seems Codecov is working on a solution to this. I don't know how they are planning to resolve this but I believe something like the workaround I used in https://github.com/codecov/feedback/issues/301#issuecomment-2009355183 can be packaged as reusable actions. The workaround works pretty well but is verbose to implement.

They could define 2 GitHub actions:

If done properly then users could just use 2 actions without extra verbosity of JS scripting.