search

codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 0 forks source link

Failed check status 17 - [RCB fully_covered_patch] changes to a PR, missing report on the base or head commits #1622

Open Adal3n3 opened 3 weeks ago

Adal3n3 commented 3 weeks ago

CI failure scenarios:

PR check status message: Your project check has failed because you have not had an uploaded report on the BASE and HEAD commits, in the case of changes to a pull request. Please ensure uploading report on the BASE and HEAD.

Use-case: Project coverage

User configurable: no

aj-codecov commented 3 weeks ago

Example scenario:

  1. I have uploaded 5 reports to my base commit
  2. I upload 1 report to head commit
  3. I have NOT covered all lines of my patch
  4. I'm seeing project coverage go WAY down
  5. AND I'm seeing my project check fail

I don't think today we can tell if this is directly responsible for failure, therefore - Suggestion:

Your project check has failed because your patch isn't fully covered. ⚠️ we've noticed your head and base commits have different numbers of coverage reports, this may be related to the failure of your project coverage check.

If we CAN say conclusively somehow in the future this is the root of the issue, we can go back to something like the below: "Your project check has failed because you have uploaded fewer coverage reports on this commit than your BASE commit. Please ensure you are uploading the same number of reports on the BASE and HEAD commits."

Adal3n3 commented 3 weeks ago

I think this is a good approach!

aj-codecov commented 2 weeks ago

We can count # of sessions per flag and note specific discrepancies, and then maybe prod a user to check their CI run to ensure it uploaded the right number of reports.

Adal3n3 commented 1 day ago

Making a note: Current we do not know the different numbers of coverage reports on head and base commits, so we don't tell customers. Fortunately, we have @giovanni-guidini is working on getting the numbers.

New design:

❌ Check status: Your project check has failed because you have 5 coverage reports on HEAD commit (482a924) and 4 coverage reports on BASE commits (7acb879). Please ensure you upload the same number of reports on the BASE and HEAD commits.

Comparison is base (7acb879) compared to head (482a924) 92.47%.

Additional details