[BUG] Issues with `codecov` on `cudf` CI

[galipremsagar]

Describe the bug There are various issues with codecov I'm trying to summarize into one master issue.

1. The codecov/patch custom github action doesn't always seem to be posted on the PR. Last PR that had this posted on was: https://github.com/rapidsai/cudf/pull/15719

   

2. codecov bot seems to have stopped commenting on PRs the diff coverage as it used to:

   

3. The base branch coverage is off by a few lines always because we generate the coverage by running the tests nightly are there any best practices and recommendations from codecov team on how to have a very upto-date base branch coverage?

[bdice]

I had a call with a couple folks from Sentry (who create Codecov), Sabiha and Vlad. We talked through a few topics related to coverage in RAPIDS.

• Codecov's carryforward option may help us with the problems we face with only running CI on a nightly basis. This may resolve (3) above. We may want to use -F <component> (like -F cudf) so that we can more easily determine coverage for each part of a repository.

• I requested a feature in Codecov. The UI for Codecov should have an option to stay in sync with the GitHub default branch (currently https://app.codecov.io/gh/rapidsai/cudf/ uses branch-23.06, which is a year old). Currently the branch must be manually adjusted, and we don't want to add work to the release process. Vlad will see what can be done on their side (a UI option to stay in sync with GitHub would be ideal).

• I reported a bug in Codecov: the "latest commit" reported for branch-24.06 on Codecov is 2 weeks old. However, there are newer coverage reports on the commit list for branch-24.06. Vlad said this seems to be a bug and they'll take a look.

• On the RAPIDS side, we're getting rate limited during Codecov uploads, which prevents us from getting full coverage reports on every nightly test run. Copying from that log below for posterity:

  [2024-05-17T07:24:36.632Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.7.2&token=*******&branch=branch-24.06&build=9124031981&build_url=https%3A%2F%2Fgithub.com%2Frapidsai%2Fcudf%2Factions%2Fruns%2F9124031981&commit=6d5f9653debe57c7eb52f42fb980d38451a9a460&job=test&pr=&service=github-actions&slug=rapidsai%2Fcudf&name=&tag=&flags=&parent=
  [2024-05-17T07:24:36.632Z] ['verbose'] Passed token was 0 characters long
  [2024-05-17T07:24:36.632Z] ['verbose'] https://codecov.io/upload/v4?package=uploader-0.7.2&branch=branch-24.06&build=9124031981&build_url=https%3A%2F%2Fgithub.com%2Frapidsai%2Fcudf%2Factions%2Fruns%2F9124031981&commit=6d5f9653debe57c7eb52f42fb980d38451a9a460&job=test&pr=&service=github-actions&slug=rapidsai%2Fcudf&name=&tag=&flags=&parent=
      Content-Type: 'text/plain'
      Content-Encoding: 'gzip'
      X-Reduced-Redundancy: 'false'
  [2024-05-17T07:24:36.844Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 429 - {'detail': ErrorDetail(string='Rate limit reached. Please upload with the Codecov repository upload token to resolve issue. Expected time to availability: 681s.', code='throttled')}
  [2024-05-17T07:24:36.846Z] ['verbose'] The error stack is: Error: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 429 - {'detail': ErrorDetail(string='Rate limit reached. Please upload with the Codecov repository upload token to resolve issue. Expected time to availability: 681s.', code='throttled')}
  at main (/snapshot/repo/dist/src/index.js)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
  [2024-05-17T07:24:36.846Z] ['verbose'] End of uploader: 532 milliseconds
  [2024-05-17T07:24:36.846Z] ['info'] Codecov will exit with status code 0. If you are expecting a non-zero exit code, please pass in the `-Z` flag

  We need to check on the GitHub application configuration and make sure it's set up correctly. We should also update to the newer Codecov CLI to upload, and use an organization token to authenticate our uploads. Currently we are using the older Codecov uploader (see the install step and shared-workflows call).

[AyodeAwe]

... We need to check on the GitHub application configuration and make sure it's set up correctly. We should also update to the newer Codecov CLI to upload, and use an organization token to authenticate our uploads. Currently we are using the older Codecov uploader (see the install step and shared-workflows call).

PRs that address this:

• https://github.com/rapidsai/ci-imgs/pull/141
• https://github.com/rapidsai/shared-workflows/pull/212

[bdice]

Amazing! Thanks @AyodeAwe. I will circle back and try this again soon. I think we may get nightly coverage reports for branch-24.06 since that's currently active in CI, but most new PRs are targeting branch-24.08 so we may be out-of-sync for a while until the 24.06 release is complete and we move our nightly CI runs to branch-24.08.