search

codecov / feedback

A place to discuss feedback about the pull request and web product experience.
37 stars 7 forks source link

Bug Report: Successful `codecovcli upload-process` failing to process by codecov #569

Open chriscerk opened 2 weeks ago

chriscerk commented 2 weeks ago

Describe the bug A clear and concise description of what the bug is.

Environment

To Reproduce Steps to reproduce the behavior:

  1. Run codecovcli --verbose upload-process

Your upload is now processing. When finished, results will be available at: OUR_ORG_LINK_TO_RESULTS Process Upload complete

  1. Wait awhile...
  2. See "Currently no uploads" in the UI and "Missing Head Report"

Expected behavior A clear and concise description of what you expected to happen.

Uploading code coverage to work.

⭐ If you have metrics on:

That would be interesting to review since we see "Currently no uploads" which according to the codecov cli should not be true.

Showing the status in the UI would be great, eg if it is queued vs. processing vs. completed

Screenshots

n/a

Additional context

This has been working fine for awhile, but since the past week this has been extremely flaky and blocking many engineering teams.

drazisil-codecov commented 2 weeks ago

Hi @chriscerk

Can you please share the full SHA?

chriscerk commented 2 weeks ago

@drazisil-codecov 0731ef732d1df7971ba12832cb27598c80434c0e is one example

drazisil-codecov commented 2 weeks ago

@chriscerk can you open a ticket and share the full CLI output from your CI? Preferably with verbose enabled? Something is not happening correctly (don't think it's anything you are doing wrong), I but I don't know what...yet.

You can mention this thread, if you want. I'll get it :)

davidlj95 commented 2 weeks ago

Same here, I can provide some examples as they're public repositories

I use Codecov GitHub Action to upload the reports there. And then have some PR checks that are mandatory to merge to the main branch. Recently (last week) I had to skip those checks given they were stuck in pending state. Investigating a bit, seems despite the action logs to upload the coverage report succeeded, no uploads appeared for that commit in Codecov's app. Hence Codecov didn't notify the results to GitHub and the check was left as pending.

An example from today:

An example from another repository, 2 days ago:

In both cases I forced a CI/CD run again by ammending the commit to change the date of the commit + force-pushing and it worked.

drazisil-codecov commented 2 weeks ago

@adrian-codecov Here are a few examples of uploads failing because create-commit didn't complete in time. Happy to huddle, I don't have any ideas for solutions at this time. Should I open an investigation ticket, or just use this one?

drazisil-codecov commented 2 weeks ago

Thanks, @davidlj95 ! Very helpful to be able to confirm that the CLI is calling all the steps in order. Definitely a server-side race condition. :-/

drazisil-codecov commented 2 weeks ago

https://github.com/codecov/feedback/issues/485 is related

davidlj95 commented 1 week ago

Today happened again. Here are the details:

An example from today:

Let me know if I'm spamming and I won't add moar comments 😛 Just providing info to confirm the issue stil exists + provide details to diagnose the issue